Cadence Documentation

The Cadence service uses specific worker roles for background processing:

• Replicator: A background worker that consumes replication tasks from remote Cadence clusters and passes them to a processor to be applied to the local Cadence cluster.
• Archiver: Handles the archival of workflow execution histories. It hosts a Cadence client worker and runs an archival system workflow. The archiver shards work across several workflows and uses the archival client to initiate archival via signal sending.

Cadence supports a session model where sessions are tied to specific resources (e.g., CPU, GPU, memory, file descriptors) rather than workers. All activities within a session are executed by the same worker that owns the resource.

Key Constraints:

• A resource must be owned by only one worker on a single host.
• Currently, a worker owns only one resource type. When creating a session, you specify the tasklist in the context, and the correct resource type is consumed automatically.
• If a worker fails, the session state transitions to Failed via background heartbeat detection.

The Scanner and Fixer workflows are data-cleanup processes used to find and resolve data inconsistencies (e.g., deleting unnecessary data, fixing bugs, or cleaning up abandoned tasklists).

• Scanner: Runs the Check method of defined Invariants. If a check fails, the result is pushed to a blobstore.
• Fixer: Runs the Fix method on the results produced by the most recent Scanner. It iterates over the scanner results in the blobstore and applies fixes.
• Invariants: Defined in common/reconciliation/invariant, these consist of Check and Fix methods. They are often managed by an InvariantManager.

Warning: These processes are considered beta-quality and are disabled by default. They can consume significant resources and modify/delete data. Use them at your own risk.

Cadence supports different domain modes to manage how workflows are distributed across clusters in a cluster group. Understanding these modes is essential for designing your multi-region architecture:

1. Local Domain: The domain is hosted by only one cluster in the group. There is no replication.
2. Active-Passive Domain: The domain is active in one cluster and passive in others. All workflows within the domain are replicated to the specified passive clusters.
3. Active-Active Domain: A domain can be active in multiple clusters simultaneously, but individual workflows are only active in exactly one cluster at any given time. This mode aims to reduce cross-region latency and data inconsistency issues compared to the Active-Passive model.

The schema/ directory contains the MongoDB schemas for all Cadence databases. The structure is organized as follows:

• schema/<database_name>/:
  • schema.json: The latest snapshot of the database schema.
  • schema.go: Golang structs representing the collection schema (used because MongoDB collections are schemaless).
  • versioned/:
    • vX.X/ (e.g., v0.1/, v1.0/): A directory for each schema version change.
      • manifest.json: A JSON file describing the schema change.
      • changes.json: A JSON file containing the specific changes for that version. Only create collection, create index, and create documents commands are permitted here.

Follow these steps to set up a local development environment featuring multiple Cadence clusters and data replication.

1. Start dependencies: Use Docker to start Cassandra.
2. Install schemas: Run the schema installation command.
3. Start Cadence servers: Launch three separate development servers, each assigned to a specific zone (xdc_cluster0, xdc_cluster1, and xdc_cluster2).
4. Register a global domain: Create a domain that replicates data across all three clusters.
5. Test with samples: Run a workflow from the Cadence Go or Java client samples.
6. Test failover: Use the CLI to update the active cluster for the domain to simulate failover and failback.

To quickly set up a local development environment with Cadence server and its dependencies (Cassandra, Prometheus, and Grafana), use Docker Compose.

Exposed Ports:

• Cadence Frontend (TChannel): 7933
• Cadence Frontend (gRPC): 7833
• Cadence Web UI: 8088
• Grafana: 3000

Note: By default, this uses the master-auto-setup image, which tracks the latest master branch changes. To update this image, run docker pull ubercadence/server:master-auto-setup.

cd $GOPATH/src/github.com/uber/cadence/docker
docker compose up

Cadence provides multiple docker-compose.yml files to enable different features or storage backends. Use the -f flag to specify the desired configuration.

Available Compose Files:

• docker-compose-es.yml: Advanced visibility with ElasticSearch 6.x
• docker-compose-es-v7.yml: Advanced visibility with ElasticSearch 7.x
• docker-compose-mysql.yml: MySQL as persistence storage
• docker-compose-postgres.yml: PostgreSQL as persistence storage
• docker-compose-statsd.yaml: Runs with Statsd+Graphite
• docker-compose-multiclusters.yaml: Runs 2 Cadence clusters
• docker-compose-custom-config.yml: Uses a custom configuration file
• docker-compose-unleash.yml: Uses OpenFeature + Unleash OSS for dynamic config

docker compose -f docker-compose-mysql.yml up

To build and run the canary binary from the project root:

1. Build the binary:

make cadence-canary

2. Start the canary worker and cron (equivalent to start -mode all):

./cadence-canary start

3. To start only the worker for manual testing:

./cadence-canary start -mode worker

4. To start only the cron canary:

./cadence-canary start -mode cronCanary

By default, it loads configuration from config/canary/development.yaml. Use ./cadence-canary -h to see available start options.

A Reset allows a workflow to go back to a previous state by forking a new branch from a specific point in its history. This is a primary tool for recovering from non-deterministic errors.

Key Behaviors:

• Starts a new run with a new runID (similar to continueAsNew).
• Collects all signals along the chain of continueAsNew from the base history.
• Warning: Resetting will cause activities, timers, or child workflows that were already completed to be re-executed. To prevent this, you can emit a signal to the workflow to mark these tasks as completed before performing the reset.
• Limitation: Resetting with child workflows is not yet fully supported.