Databricks AI Dev Kit

The Databricks Builder App is a web application providing a Claude Code agent interface designed for building on Databricks. It uses a 'Skills' system to supply product-specific workflows (such as jobs, pipelines, SQL, and Unity Catalog operations). Claude executes these workflows using either the authenticated Databricks CLI or short Python SDK scripts.

Security Warning: This application wraps Claude Code. Projects created by different users are not strongly isolated (e.g., no microVM or Docker isolation for Claude sessions). Only grant access to trusted users.

Depending on your goal, you can use different components of the Databricks AI ecosystem:

• Install Skills: The recommended starting point to add Databricks skills to your user or existing project folder.
• Genie Code Skills: For uploading selected skills into your workspace specifically for Genie Code.
• Visual Builder App: A web-based UI for Databricks development (located in databricks-builder-app/).
• MCP Tools: A standalone Model Context Protocol (MCP) server that exposes Databricks actions to AI clients. While skills are generally recommended, this is useful if a custom MCP server is required.

All API calls made through the MCP server are tagged with a custom User-Agent header, allowing you to filter calls in the system.access.audit system table. The project name is automatically detected from the git remote URL.

Header Format: databricks-ai-dev-kit/0.1.0 databricks-sdk-py/... project/<auto-detected-repo-name>

Note: Audit log entries may take 2–10 minutes to appear. The workspace must have Unity Catalog enabled to query system.access.audit.

Configuring Context

You can provide additional context to the agent via optional parameters in stream_agent_response to guide its execution:

• warehouse_id: Default SQL warehouse.
• cluster_id: Default cluster for Python execution.
• default_catalog: Default Unity Catalog.
• default_schema: Default schema.
• workspace_folder: Workspace folder for file uploads.

Session Management

To resume a conversation, capture the session_id from a result event and pass it into subsequent calls to stream_agent_response.

# Resume conversation using session_id
async for event in stream_agent_response(
    project_id="demo",
    message="What did I just ask?",
    session_id=session_id,  # Pass the ID captured from a previous 'result' event
):
    ...

The application architecture consists of three main layers:

1. Web Application: A React frontend (Chat UI, Project Selector, Conversation List) communicating via SSE with a FastAPI backend (endpoints: /api/invoke_agent, /api/projects, /api/conversations).
2. Claude Code Session: Each user message spawns a session via claude-agent-sdk. The session has access to:
   • Built-in Tools: Read, Write, Edit, Bash, Glob, Grep, and Skill.
   • Skills: CLI or Python SDK workflows for Databricks products (jobs, pipelines, SQL, UC, etc.).
   • Context: Uses project-scoped .databrickscfg for the Databricks CLI/WorkspaceClient and mcp_servers={}.
3. Databricks Workspace: The target environment containing SQL Warehouses, Clusters, Unity Catalog, and Workspace assets.

Databricks provides two primary paths for AI-assisted coding depending on your preferred environment:

1. Genie Code: A free, first-party AI coding experience built directly into the Databricks workspace. It has native access to your notebooks, jobs, and Unity Catalog data. Best for users working primarily within the Databricks UI.
2. AI Dev Kit: Brings Databricks expertise, tutorials, and official agent skills to your existing local IDE or AI coding agent (e.g., Claude Code, Cursor, GitHub Copilot, Gemini CLI, Windsurf, etc.). Best for developers who want to build on Databricks using their preferred local development tools.

The Builder App operates in two distinct modes based on the presence of the DATABRICKS_CLIENT_ID environment variable. This distinction changes how Claude configuration is handled and how authentication is performed.

Local Mode

• Detection: DATABRICKS_CLIENT_ID is unset.
• Claude Config: Uses the default ~/.claude directory.
• Authentication: Uses ANTHROPIC_API_KEY and ANTHROPIC_BASE_URL via _build_claude_auth.
• Warning: Never set CLAUDE_CONFIG_DIR in local mode, as it will disconnect claude login / Keychain credentials and break local chat functionality.

Deployed Mode (Apps)

• Detection: DATABRICKS_CLIENT_ID is set (detected via is_deployed_mode()).
• Claude Config: Uses CLAUDE_CONFIG_DIR=<project>/.claude.
• Authentication: Uses project apiKeyHelper, .anthropic_token, and .claude/settings.json.

The server uses the Databricks Unified Authentication chain. To ensure your MCP client can authenticate with your workspace, you must provide credentials via environment variables in the client's MCP configuration env block.

Option 1: Named profile (Recommended)

If you have a profile configured in ~/.databrickscfg, use: DATABRICKS_CONFIG_PROFILE="your-profile"

Option 2: Explicit host and token

If you do not use profiles, provide the host and token directly: DATABRICKS_HOST="https://your-workspace.cloud.databricks.com" DATABRICKS_TOKEN="your-token"

Projects are stored on the local filesystem and are automatically backed up to a PostgreSQL database (Lakebase) to ensure durability.

Project Directory Structure

projects/
  <project-uuid>/
    .claude/
      skills/        # Copied skills for this project
    src/             # User's code files
    ...

Backup Process

• Trigger: The project is marked for backup after every agent interaction.
• Frequency: A background worker runs every 10 minutes.
• Storage: Projects are zipped and stored in PostgreSQL.
• Restoration: If a project is missing from the local filesystem, it is automatically restored from the PostgreSQL backup upon access.