12-Factor Agents Framework

For production workloads, avoid 'synchronous mode' where the server polls for human responses. Instead, implement an asynchronous pattern using webhooks to 'launch, pause, and resume' agent execution.

Architecture Pattern:

1. Initialize HumanLayer in your server.
2. Asynchronous /thread endpoint: When a request comes in, the endpoint should start processing and return a response immediately (e.g., including a state.threadId).
3. Pause Execution: If the agent needs human input (e.g., for request_more_information or done_for_now), create a human contact and end the current processing loop.
4. Webhook Endpoint: Implement a /webhook endpoint to receive the human's response from HumanLayer.
5. Resume Execution: Use a handleHumanResponse function to process the incoming webhook and resume the agent's logic.

This approach follows the principle of launching/pausing/resuming with simple APIs, ensuring the server isn't sitting in a blocking loop while waiting for a human.

Avoid using 'black box' agent frameworks that hide prompt engineering behind high-level abstractions like role, goal, or personality. While these are helpful for prototyping, they make it difficult to tune specific tokens or reverse-engineer the exact instructions being sent to the model.

Instead, treat prompts as first-class code. This means explicitly defining the system and user instructions, often using templating tools or domain-specific languages (DSLs) like BAML. This approach allows you to treat the prompt as the primary interface between your application logic and the LLM.

Benefits of owning your prompts:

• Full Control: Write exact instructions without hidden abstractions.
• Testing and Evals: Apply standard software testing and evaluation practices to your prompts.
• Iteration: Modify prompts quickly based on real-world performance data.
• Transparency: Maintain a clear understanding of exactly what instructions the agent is receiving.
• Role Hacking: Leverage non-standard usage of user/assistant roles or specific API behaviors (e.g., model gaslighting or legacy completion APIs).

# AVOID THIS 'BLACK BOX' PATTERN:
agent = Agent(
  role="...",
  goal="...",
  personality="...",
  tools=[tool1, tool2, tool3]
)

task = Task(
  instructions="...",
  expected_output=OutputModel
)

result = agent.run(task)

A 12-factor agent relies on a Thread to manage state and an agentLoop to process turns.

• Thread: A class that holds an array of Event objects. It includes a serializeForLLM() method to convert the conversation history into a format (like JSON) suitable for the LLM prompt.
• agentLoop: An async function that takes a Thread, serializes it, calls the BAML-generated client function (e.g., b.DetermineNextStep), and returns the structured response.
• Event: A type representing a single interaction, typically containing a type (e.g., user_input) and data.

export interface Event {
    type: string
    data: any;
}

export class Thread {
    events: Event[] = [];
    constructor(events: Event[]) { this.events = events; }

    serializeForLLM() {
        return JSON.stringify(this.events);
    }
}

export async function agentLoop(thread: Thread): Promise<AgentResponse> {
    const nextStep = await b.DetermineNextStep(thread.serializeForLLM());
    return nextStep;
}

To build reliable AI agents, aim to unify execution state and business state into a single source of truth. This reduces complexity and makes the agent's lifecycle easier to manage.

Definitions

• Execution state: Metadata about the agent's current progress, such as the current step, next step, waiting status, or retry counts.
• Business state: The actual content of the workflow, such as the list of OpenAI messages, tool calls, and tool results.

The Goal: State as Context

Whenever possible, engineer your application so that the execution state can be inferred directly from the context window. While some data (like session IDs or password contexts) must remain outside the context window, your goal should be to minimize these external dependencies by embracing the principle of owning your context window.

Benefits of Unified State

• Simplicity: You maintain only one source of truth for all state.
• Serialization: The entire agent thread becomes trivially serializable and deserializable.
• Debugging: The complete history of what happened and why is visible in one place.
• Flexibility: You can add new state information simply by introducing new event types.
• Recovery: You can resume an agent from any point by simply loading the thread.
• Forking: You can fork a thread at any point by copying a subset of the thread into a new context or state ID.
• Observability: It is easy to convert a unified thread into human-readable markdown or a rich Web UI.

To build more efficient agents, avoid prompting the model to call tools that you can predict it will need. Instead of waiting for the model to emit a tool-call intent (which costs a round trip of tokens), call the tool deterministically in your application code and include the results in the context window before the model even makes its first decision.

There are two primary patterns for implementing this:

1. Explicit Parameter Injection: Fetch the data and pass it as a direct argument to your determine_next_step function (or equivalent decision-making call). This is useful when the data is highly relevant to the immediate decision.
2. Thread/Event Injection: Fetch the data and append it to your thread or events list as a completed event. This allows you to keep your decision-making function signature simple and relies on the model's ability to process the history of the conversation/execution.

Core Principle: If you already know what tools you'll want the model to call, call them DETERMINISTICALLY and let the model do the hard part of figuring out how to use their outputs.

# Pattern 1: Explicit Parameter Injection
thread = {"events": [initial_message]}
git_tags = await fetch_git_tags()

# Pass the pre-fetched data directly to the decision function
next_step = await determine_next_step(thread, git_tags)

while True:
    match next_step.intent:
        case 'deploy_backend_to_prod':
            deploy_result = await deploy_backend_to_prod(next_step.data.tag)
            thread["events"].append({
                "type": 'deploy_backend_to_prod',
                "data": deploy_result,
            })
        case 'done_for_now':
            await notify_human(next_step.message)
            break

Most LLM clients use a standard message-based format consisting of system, user, assistant, and tool roles. While convenient, this format may not be the most token- or attention-efficient for complex agents.

An alternative is to build a custom context format—such as using XML-style tags—and pack the entire history into a single user message. This allows you to structure events (like Slack messages, tool intents, and tool results) in a way that maximizes the LLM's ability to parse the sequence of events.

[
  {
    "role": "system",
    "content": "You are a helpful assistant..."
  },
  {
    "role": "user",
    "content": |
            Here's everything that happened so far:
        
        <slack_message>
            From: @alex
            Channel: #deployments
            Text: Can you deploy the backend?
        </slack_message>
        
        <list_git_tags>
            intent: "list_git_tags"
        </list_git_tags>
        
        <list_git_tags_result>
            tags:
              - name: "v1.2.3"
                commit: "abc123"
                date: "2024-03-15T10:00:00Z"
        </list_git_tags_result>
        
        what's the next step?
    }
]