NiPyAPI Python Client SDK

repository·main·Indexed 42 days ago

https://github.com/chaffelson/nipyapi

A rich Python Client SDK for Apache NiFi featuring a high-level SDK for common tasks, a low-level SDK for full API access, and a CLI for CI/CD pipelines. It supports multiple authentication profiles including single-user, secure-ldap, secure-mtls, and secure-oidc, and provides tools for managing flow development lifecycles and NiFi Registry version control.

Tokens
66.1K
Snippets
199
Records
261
Agent score
55%

What's inside NiPyAPI

  1. Explore NiPyAPI Registry API classes

    main

    NiPyAPI provides 13 specialized Registry API classes to interact with specific NiFi Registry REST endpoints. Each class encapsulates methods for managing different Registry resources. To interact with the Registry, you must identify the specific API class corresponding to the resource you wish to manage (e.g., FlowsApi for flows, BucketsApi for buckets, or UsersApi for users).

    Available Registry API classes include:
    - about_api
    - access_api
    - bucket_bundles_api
    - bucket_flows_api
    - buckets_api
    - bundles_api
    - config_api
    - extension_repository_api
    - extensions_api
    - flows_api
    - items_api
    - policies_api
    - tenants_api
  2. Explore NiFi API classes in NiPyAPI

    main

    NiPyAPI provides a comprehensive client for the NiFi REST API, organized into 28 distinct API classes. Each class corresponds to specific NiFi endpoints and provides specialized methods for interacting with those resources.

    Available API classes include:

    • access_api
    • authentication_api
    • connections_api
    • controller_api
    • controller_services_api
    • counters_api
    • data_transfer_api
    • flow_api
    • flow_file_queues_api
    • funnels_api
    • input_ports_api
    • labels_api
    • output_ports_api
    • parameter_contexts_api
    • parameter_providers_api
    • policies_api
    • process_groups_api
    • processors_api
    • provenance_api
    • provenance_events_api
    • remote_process_groups_api
    • reporting_tasks_api
    • resources_api
    • site_to_site_api
    • snippets_api
    • system_diagnostics_api
    • tenants_api
    • versions_api
  3. NiPyAPI Python SDK Modules Overview

    main

    NiPyAPI provides different levels of abstraction depending on your needs:

    • Mid-level Client SDK: High-level functions for common tasks. Found in modules like nipyapi.canvas, nipyapi.security, nipyapi.parameters, and nipyapi.versioning.
    • Low-level Client SDKs: Full API implementation of NiFi and NiFi Registry. Found in nipyapi.nifi and nipyapi.registry.
    • CI/CD Operations: Specialized commands for deployment pipelines found in nipyapi.ci.
  4. Handle NAR deletion and component protection

    main

    NiFi prevents the deletion of NARs that are currently being used by active components (processors, controller services, etc.).

    Deletion Protection

    If you attempt to delete a NAR in use without forcing it, nipyapi.extensions.delete_nar will raise a ValueError with a message indicating components are instantiated from this NAR.

    Force Deletion and Orphaned Components

    If you use force=True to delete a NAR while components are using it, those components become orphaned. Their state will change to STOPPED or INVALID with a "Missing Processor" error.

    Recovering Orphaned Components

    To recover components orphaned by a force-deletion, re-upload the exact same NAR version. The components will automatically recover, and their properties and configurations will be preserved.

    ActionResultComponent State
    Delete NAR (no force)BlockedUnchanged
    Delete NAR (force=True)NAR removedSTOPPED, INVALID with "Missing Processor" error
    Re-upload same NARNAR installedRecovers to VALID, properties preserved
    import nipyapi
    
    # Attempting to delete a NAR in use
    try:
        nipyapi.extensions.delete_nar(nar_id)
    except ValueError as e:
        # "Unable to delete NAR [...] because components are instantiated from this NAR"
        print(f"NAR protected: {e}")
    
    # Recovering orphaned components by re-uploading
    nar = nipyapi.extensions.upload_nar("/path/to/my-processor.nar")
    
    # Wait for processor to recover
    proc = nipyapi.extensions.wait_for_processor_init(processor_id)
    print(f"Recovered: {proc.component.validation_status}")
  5. Use Base vs. HTTP Info methods in Generated APIs

    main

    When using the low-level Generated APIs, every operation provides two distinct methods. You should choose based on how much metadata you require:

    • Base Methods (e.g., copy()): Return only the response data directly. Use these for standard operations where you only care about the payload.
    • HTTP Info Methods (e.g., copy_with_http_info()): Return a tuple containing the response data, the HTTP status code, and the HTTP headers. Use these when you need to inspect HTTP metadata or debug error details.
    import nipyapi
    
    # Low-level API approach
    api_instance = nipyapi.nifi.ProcessGroupsApi()
    
    # 1. Get just the data (Base Method)
    flow = api_instance.get_flow('root')
    
    # 2. Get data + HTTP details (HTTP Info Method)
    flow, status, headers = api_instance.get_flow_with_http_info('root')
    print(f"HTTP Status: {status}")
  6. Configure NiFi Registry Client with Explicit SSL Context Services

    main

    When NiFi and the Registry use different certificate authorities, or when you have specific PKI requirements (like Enterprise PKI or component-specific certificates), you must use Explicit SSL Context Services.

    To do this:

    1. Create an SSL Context Service using nipyapi.security.ensure_ssl_context() providing the keystore and truststore files and passwords.
    2. Pass the resulting SSL context object to the ssl_context_service parameter when calling nipyapi.versioning.ensure_registry_client().
    # Create SSL Context Service for specific PKI requirements
    ssl_context = nipyapi.security.ensure_ssl_context(
        name='registry_ssl_context',
        parent_pg=parent_pg,
        keystore_file='/path/to/specific/keystore.p12',
        keystore_password='password',
        truststore_file='/path/to/specific/truststore.p12',
        truststore_password='password'
    )
    
    # Registry Client with explicit SSL Context
    registry_client = nipyapi.versioning.ensure_registry_client(
        name='secure_registry_client',
        uri='https://secure-registry.example.com/nifi-registry-api',
        description='Registry Client with explicit SSL Context',
        ssl_context_service=ssl_context
    )
  7. Follow Google-style docstring standards

    main

    NiPyAPI uses Google-style docstrings for both Sphinx documentation and CLI help generation. To ensure compatibility with both:

    • Summary: Use triple double-quotes. The first line must be a concise imperative summary.
    • Sections: Include Args:, Returns:, and Raises: sections where applicable.
    • Types: Do not duplicate type hints in docstrings; focus on semantics and constraints. For union types, use the (type1 or type2) format.
    • CLI Compatibility: Do NOT use nested bullet lists under Args parameters, as python-fire will truncate the description in the CLI help text. Use inline formatting instead.
    • Examples: Use the Example:: (singular) syntax. You must include a blank line before the code block for Sphinx to render it correctly.
    • Cross-references: Use Sphinx notation like :class:~nipyapi.module.ClassName`` to create clickable links. The ~ prefix shows only the class name.
    def get_process_group(pg_id, identifier_type='id'):
        """Return a specific process group by identifier.
    
        Args:
            pg_id (str): The identifier of the process group
            identifier_type (str): 'id' or 'name'
    
        Returns:
             :class:`~nipyapi.nifi.models.ProcessGroupEntity`: The matching
             process group, or None if not found
    
        Raises:
            ValueError: If identifier_type is not 'id' or 'name'
        """
        pass
    
    def my_function():
        """Do something useful.
    
        Example::
    
            result = my_function()
            print(result)
        """
        pass
  8. Handle NiPyAPI CLI Errors in Scripts

    main

    The CLI uses exit codes to signal success or failure, making it reliable for shell scripting.

    • 0: Success (no error or errors key in the result).
    • 1: Error (either a Python exception was raised, or the result contains an error or errors key).

    When an operation fails (especially in ci modules), the JSON response will include an error (string) or errors (pipe-separated string) field.

    # Script can rely on exit code
    if nipyapi ci verify_config --process_group_id "$PG_ID"; then
        nipyapi ci start_flow --process_group_id "$PG_ID"
    else
        echo "Verification failed, not starting flow"
        exit 1
    fi
  9. How Python processor initialization works

    main

    Python processors require a unique initialization lifecycle because they must set up a virtual environment and download dependencies before they are ready for configuration.

    Initialization States

    • initializing: Setting up the Python virtual environment.
    • downloading_dependencies: Downloading packages from PyPI.
    • dependency_failed: Failed to download dependencies (e.g., network issues).
    • ready: Environment is set up, properties are loaded, and the processor is ready for configuration.
    • missing_nar: The NAR was removed, leaving the processor orphaned.
    • error: An unknown error occurred.

    Best Practices

    • Air-gapped environments: If NiFi cannot reach PyPI, bundle all dependencies inside the NAR file itself.
    • Caching: NiFi caches virtual environments based on the NAR coordinate (group:artifact:version). A version bump (e.g., 1.0.0 to 1.0.1) always triggers a fresh environment creation and full dependency download.
    import nipyapi
    
    # Monitor initialization status
    status = nipyapi.extensions.get_processor_init_status(processor_id)
    
    print(f"Status: {status['status']}")
    print(f"Ready: {status['is_ready']}")
    print(f"Has Properties: {status['has_properties']}")
    print(f"Message: {status['init_message']}")
    
    if status['validation_errors']:
        for error in status['validation_errors']:
            print(f"  - {error}")
  10. Configure NiFi Registry Client with Implicit SSL Trust

    main

    If NiFi and the NiFi Registry share a common root Certificate Authority (CA), you can use Implicit Trust. In this mode, when the Registry Client's ssl-context-service property is null, NiFi falls back to its global SSL configuration. This is the simplest configuration and requires no additional SSL Context Services in NiFi.

    # No SSL Context Service needed - uses NiFi's global SSL configuration
    registry_client = nipyapi.versioning.ensure_registry_client(
        name='my_registry_client',
        uri='https://registry.example.com/nifi-registry-api',
        description='Registry Client using implicit SSL trust'
    )