repository·main·Indexed 12 days ago
https://github.com/temporalio/temporalA durable execution platform for building scalable and reliable applications by automatically managing Workflow state and execution. Includes documentation on the Matching Client's partition selection and routing (Basic and Spread Routing), Google Cloud Storage (gstorage) archival configuration, and guides for implementing custom HistoryArchiver and VisibilityArchiver interfaces.
The docs/ directory contains documentation specifically for those working closely with the Temporal server.
/architecture section for high-level explanations and diagrams of Temporal's core concepts and technological underpinnings./development section for instructions on setting up a local development environment and advanced topics like adding migrations or new RPCs./admin section for operational guides. Note that current production deployment, configuration, and monitoring guidance is primarily hosted on the official Temporal docs website: https://docs.temporal.io/self-hosted-guide.Mutable State is a collection of persisted data structures that summarize the current state of a Workflow Execution (e.g., identities of in-progress activities, timers, and child workflows).
While this state can be recomputed from the full Workflow History, storing it as Mutable State allows for much faster request handling.
Key characteristics:
MutableStateImpl struct which implements the MutableState interface.CHASM (Coordinated Heterogeneous Application State Machines) is a framework designed for high-scale, low-latency state management. While Temporal Workflows are powerful, they can be heavyweight for certain use cases (e.g., handling millions of signals or very large payloads).
CHASM treats a Workflow as just one type of Application State Machine (ASM). An ASM leverages Temporal's underlying infrastructure—such as sharding, routing, atomic storage, and failure recovery—but provides a lighter, typed API that hides distributed systems complexity. This makes it suitable for problems where a full Workflow might be too slow or complex.
Nexus RPC is a service framework designed for long-running, arbitrary-length operations that may extend beyond the lifetime of a traditional RPC. It serves as a connectivity layer for durable executions within and across namespaces, clusters, and regions.
Services can expose two types of operations:
Workflow Execution History is a linear sequence of History Events that defines the state of a Workflow Execution.
Reset or subject to conflict resolution.Nexus uses state machines to manage the lifecycle of operations and callbacks, ensuring reliability through retries.
Managed by the nexusoperations component. An operation transitions through states like Scheduled $\rightarrow$ BackingOff $\rightarrow$ Started $\rightarrow$ Succeeded/Failed/Canceled/TimedOut.
component.nexusoperations.limit.scheduleToCloseTimeout.component.nexusoperations.request.timeout (default: 10s).Callbacks provide a reliable mechanism for delivering workflow outcomes (success, failure, cancellation, or termination).
workflow closed.Callback state machine moves from Standby $\rightarrow$ Scheduled $\rightarrow$ BackingOff $\rightarrow$ Succeeded/Failed.component.callbacks.request.timeout (default: 10s).CHASM uses logical clocks to provide total ordering of state changes across distributed systems.
Every transition is stamped with a VersionedTransition, which includes:
FailoverVersion: Increments during cross-DC failover.TransitionCount: Increments with every state update within the Execution.When a component starts an external task and requires a callback, it creates a ComponentRef. This is a serialized token containing:
ExecutionKey and ComponentPath: To route the callback to the correct node.initialVT: The VersionedTransition when the node was created (guards against hitting a deleted/recreated node).lastUpdateVT: The VersionedTransition when the token was issued (guards against stale callbacks updating a node that has already moved past the expected state).Consistency Check: When a callback arrives, the Engine verifies both initialVT and lastUpdateVT. If there is a mismatch, the callback is rejected.
The outbound task queue is an internal, sharded, "immediate" queue within the history service used to target external destinations. It is designed to handle long-running external requests (up to 10 seconds) without blocking other operations.
Isolation Mechanisms:
history.outboundQueueMaxReaderCount.history.outboundQueue.groupLimiter.bufferSize.history.outboundQueue.groupLimiter.concurrency.history.outboundQueue.hostScheduler.maxTaskRPS.DestinationDownErrors (retryable HTTP errors or timeouts) to prevent overwhelming a failing destination. Configured via history.outboundQueue.circuitBreakerSettings.Disabling the Processor: You can disable the outbound queue reader by setting history.outboundTaskBatchSize to 0.
The sequencing_id field is used to specify the order in which messages must be processed relative to existing events and commands.
Supported values:
event_id: Indicates the specific event after which the message should be processed by the worker.command_index: Indicates the specific command after which the message should be processed by the server.Current Implementation Details:
event_id to the ID preceding the WorkflowTaskStartedEvent. This effectively means all messages are processed after all events due to how the server handles the reorder buffer.command_index directly. Instead, they use a special command of type COMMAND_TYPE_PROTOCOL_MESSAGE to indicate where a message should be processed. If an Update is rejected, this command is not added, and the server assumes the message is rejected.