DOCSEARCHbyAlgolia
Submit Docs

Cloudify Manager

repository·master·Indexed 41 days ago

https://github.com/cloudify-cosmo/cloudify-manager

A REST-based management service for orchestrating cloud resources and workflows. It consists of a core REST API service, pre-defined system workflows, and an integration testing suite. Key components include an execution scheduler for workflow automation and a specialized AMQP-Postgres transport for consuming logs and events from RabbitMQ into PostgreSQL.

Tokens
4.1K
Snippets
17
Records
24
Agent score
100%

What's inside Cloudify Manager

  1. Overview of Cloudify Manager service containers

    master
    The service_containers directory contains the Dockerfiles, configuration files, and build scripts required to create Docker images for Cloudify Manager service components. These containers are specifically intended for services that are based on external software rather than being generated directly from Cloudify code.

  3. Overview of Cloudify Manager components

    master

    Cloudify Manager is composed of three primary components:

    1. Cloudify Manager REST service: The core service providing the REST API for managing Cloudify resources and workflows.
    2. Cloudify system-workflows: Pre-defined workflows used by the system.
    3. Integration tests: A suite of tests used to verify the interaction between components.

  5. Manage schedule concurrency with `scheduler_lock`

    master

    To prevent multiple manager instances from running the same schedule simultaneously, the scheduler uses a locking mechanism based on try_acquire_lock_on_table.

    Locking Mechanism:

    • Each schedule uses a unique lock number calculated as SCHEDULER_LOCK_BASE + schedule._storage_id (where SCHEDULER_LOCK_BASE is 10000).
    • The scheduler_lock(lock_number) context manager attempts to acquire the lock. If try_acquire_lock_on_table returns False, the execution is skipped to avoid conflicts.
    • The lock is automatically released via unlock_table(lock_number) when the context manager exits.
    @contextmanager
def scheduler_lock(lock_number):
    locked = try_acquire_lock_on_table(lock_number)
    try:
        yield locked
    finally:
        if locked:
            unlock_table(lock_number)

  6. How the scheduler loop and timing work

    master

    The scheduler operates using a main() loop that utilizes a LoopTimer context manager to maintain a consistent execution interval.

    The Lifecycle:

    1. Loop Start: The main() function enters a while True loop.
    2. Timing: The LoopTimer records the start time. Upon exiting the with block, it calculates seconds_wait by subtracting the elapsed time from the defined interval (defaulting to 60 seconds).
    3. Schedule Check: check_schedules() is called to find enabled schedules where next_occurrence < datetime.utcnow().
    4. Maintenance Check: If the system is in MAINTENANCE_MODE_ACTIVATED, schedules are skipped.
    5. Execution: For each valid schedule, try_run(schedule) attempts to acquire a distributed lock and execute the workflow.
    6. Sleep: The loop sleeps for the calculated t.seconds_wait to ensure the next check happens at the correct interval.
    # Conceptual loop structure used by the scheduler
while True:
    with LoopTimer(interval=60) as t:
        check_schedules()
    time.sleep(t.seconds_wait)

  7. Handle Database Failover in Cloudify Manager

    master

    Cloudify Manager includes built-in mechanisms to handle database failovers (e.g., when a PostgreSQL standby becomes the leader or a proxy restarts).

    Automatic Failover Detection

    • Connection Level: The handle_db_failover function listens for SQLAlchemy Pool close events to trigger update_db_uri().
    • Request Level: The cope_with_db_failover function runs before every request. It checks if the current database is in recovery mode (SELECT pg_is_in_recovery()). If it is a standby, it updates the DB URI, closes all sessions, and retries.

    Manual URI Update

    If you need to programmatically update the database connection string within the app context, use update_db_uri(). This method synchronizes the SQLAlchemy configuration with the current config.instance.db_url and logs changes to the database host.

  8. Configure Debug Environment via Environment Variables

    master

    The server can be configured to connect to a remote debugger (like pydevd) using the DEBUG_CONFIG environment variable.

    1. Set DEBUG_CONFIG to the path of a YAML file.
    2. The YAML file must contain:
      • is_debug_on: true
      • host: "<debug_host>"

    When CloudifyFlaskApp is initialized, it will attempt to connect to the specified host on port 53100.

    export DEBUG_CONFIG="/path/to/debug_config.yaml"
# debug_config.yaml content:
# is_debug_on: true
# host: "127.0.0.1"

  9. Configure Stage Backend and Composer Backend REST address

    master

    Both stage_backend and composer_backend services use the RESTSERVICE_ADDRESS environment variable to identify the address of the REST service (typically the nginx container).

    Example configuration:

    stage_backend:
    environment:
      RESTSERVICE_ADDRESS: nginx
    stage_backend:
    environment:
      RESTSERVICE_ADDRESS: nginx

composer_backend:
    environment:
      RESTSERVICE_ADDRESS: nginx

  10. Configure PostgreSQL environment variables

    master

    When using the postgresql service in the docker-compose.yml setup, you can configure the database credentials and name using the following environment variables:

    • POSTGRES_DB: The name of the database to create.
    • POSTGRES_USER: The username for the database.
    • POSTGRES_PASSWORD: The password for the database.

    Default values in the provided configuration are cloudify_db, cloudify, and cloudify respectively.

    postgresql:
    container_name: postgresql
    image: postgres
    environment:
      POSTGRES_DB: cloudify_db
      POSTGRES_USER: cloudify
      POSTGRES_PASSWORD: cloudify

  12. Configure MinIO (fileserver) credentials and ports

    master

    The fileserver service uses MinIO. You can configure access via the following environment variables and ports:

    Environment Variables:

    • MINIO_ROOT_USER: The root user for MinIO.
    • MINIO_ROOT_PASSWORD: The root password for MinIO.

    Ports:

    • 9000: MinIO API port.
    • 9090: MinIO Console port.

    Default values are admin for the user and admin123 for the password.

    fileserver:
    image: minio/minio
    environment:
      MINIO_ROOT_USER: admin
      MINIO_ROOT_PASSWORD: admin123
    ports:
      - "9000:9000"
      - "9090:9090"