Tinker Python SDK

The SamplingClient is used to generate text tokens from either a base model or weights saved via a TrainingClient.

Obtaining a Client

You typically obtain a SamplingClient through one of two methods:

1. service_client.create_sampling_client(): To use a base model.
2. training_client.save_weights_and_get_sampling_client(): To use specific trained weights.

Multi-processing and Subprocess Isolation

• Multi-processing: SamplingClient is picklable and safe to pass to multiple processes/workers. However, you should always create the client in the main process and then pass it to workers. ServiceClient and TrainingClient must remain managed in the main process.
• Subprocess Isolation: To prevent CPU-heavy user code (like grading or environment interactions) from stalling networking IO and heartbeats due to GIL contention, set the environment variable TINKER_SUBPROCESS_SAMPLING=1. This runs sample() and compute_logprobs() in a dedicated subprocess transparently.

sampling_client = service_client.create_sampling_client(base_model="Qwen/Qwen3-8B")
prompt = types.ModelInput.from_ints(tokenizer.encode("The weather today is"))
params = types.SamplingParams(max_tokens=20, temperature=0.7)
future = sampling_client.sample(prompt=prompt, sampling_params=params, num_samples=1)
result = future.result()

To resume training from saved weights, you can use two different methods depending on whether you need to restore the optimizer state:

1. create_training_client_from_state(path): Loads only the model weights. The optimizer state is reset. Use this to continue training from a specific weight checkpoint without necessarily needing the previous momentum.
2. create_training_client_from_state_with_optimizer(path): Loads both model weights and the optimizer state (e.g., Adam momentum). Use this to resume training exactly where it left off for maximum continuity.

# Option 1: Weights only, optimizer resets
training_client = service_client.create_training_client_from_state(
    "tinker://run-id/weights/checkpoint-001"
)

# Option 2: Full state restoration (weights + optimizer momentum)
training_client = service_client.create_training_client_from_state_with_optimizer(
    "tinker://run-id/weights/checkpoint-001"
)

The RestClient is used for REST API operations such as listing checkpoints, retrieving metadata, and managing training runs. You typically obtain an instance by calling service_client.create_rest_client().

Key capabilities include:

• Listing checkpoints (training or sampler) and user checkpoints.
• Retrieving training run information and metadata.
• Managing checkpoint visibility (publishing/unpublishing) and TTL (Time To Live).
• Downloading checkpoint archives via signed URLs.
• Deleting checkpoints.

rest_client = service_client.create_rest_client()
training_run = rest_client.get_training_run("run-id").result()
print(f"Training Run: {training_run.training_run_id}, LoRA: {training_run.is_lora}")
checkpoints = rest_client.list_checkpoints("run-id").result()
print(f"Found {len(checkpoints.checkpoints)} checkpoints")
for checkpoint in checkpoints.checkpoints:
    print(f"  {checkpoint.checkpoint_type}: {checkpoint.checkpoint_id}")

An APIFuture is an abstract base class for handling asynchronous operations that provides a unified interface for both synchronous and asynchronous access. This allows you to use the same object in different execution contexts.

• In an async context: You can await the future directly or use await future.result_async().
• In a sync context: You can call future.result(), which will block the current thread until the operation completes.

This pattern is useful when an API method returns a handle to a long-running task, and you need the flexibility to either wait for it immediately or integrate it into an asyncio event loop.

# In async context
future = training_client.forward_backward(data, "cross_entropy")
result = await future  # Or await future.result_async()

# In sync context
future = training_client.forward_backward(data, "cross_entropy")
result = future.result()

Save weights

Use save_state(name, ttl_seconds=None) to save model weights to persistent storage. Returns an APIFuture containing the checkpoint path.

Load weights (weights only)

Use load_state(path) to load model weights from a Tinker path (e.g., "tinker://run-id/weights/checkpoint-001"). This does not restore optimizer state (e.g., Adam momentum), so the optimizer will reset.

Load weights and optimizer state

Use load_state_with_optimizer(path) to restore both model weights and the optimizer state. This is required to resume training with the same momentum and optimizer trajectory.

# Save after training
save_future = training_client.save_state("checkpoint-001")
result = await save_future
print(f"Saved to: {result.path}")

# Resume training with restored optimizer momentum
load_future = training_client.load_state_with_optimizer(
    "tinker://run-id/weights/checkpoint-001"
)
await load_future

To use trained weights for inference, you can either save them specifically for a sampler or create a client directly from the current state.

Save specifically for a sampler

Use save_weights_for_sampler(name, ttl_seconds=None) to get a path specifically intended for a SamplingClient.

Create a SamplingClient from a path

Use create_sampling_client(model_path, retry_config=None) with a saved weight path.

Direct conversion

Use save_weights_and_get_sampling_client(name=None, retry_config=None) to save the current weights and immediately return a SamplingClient configured with them.

# Option 1: Save and then create client
save_future = training_client.save_weights_for_sampler("sampler-001")
result = await save_future
sampling_client = service_client.create_sampling_client(model_path=result.path)

# Option 2: Direct conversion
sampling_client = training_client.save_weights_and_get_sampling_client()

# Use for inference
prompt = types.ModelInput.from_ints(tokenizer.encode("Hello"))
params = types.SamplingParams(max_tokens=20)
result = sampling_client.sample(prompt, 1, params).result()

The APIFuture class provides a unified interface for handling asynchronous operations that can be accessed in both synchronous and asynchronous contexts. This is the standard way to handle results from Tinker API methods that involve long-running or concurrent tasks.

Asynchronous Usage

In an async context, you can await the future directly or use the result_async() method. Awaiting the object directly is equivalent to calling await future.result_async().

# In async context
future = training_client.forward_backward(data, "cross_entropy")
result = await future  # Or await future.result_async()

Synchronous Usage

In a synchronous context, use the result() method. This call will block the current thread until the operation completes or the timeout is reached.

# In sync context
future = training_client.forward_backward(data, "cross_entropy")
result = future.result()