DOCSEARCHbyAlgolia
Submit Docs

Foundation Lab Framework

repository·main·Indexed 22 days ago

https://github.com/rudrankriyam/foundation-models-framework-lab

A native iOS and macOS workbench for learning, testing, and shipping with Apple's Foundation Models framework. It includes Swift playgrounds for session management, sampling control, and tool use, as well as the FoundationLabCore package for shared logic. The ecosystem features the foundation-models-adapter-studio (fmas) CLI for adapter training and export, and the afm CLI for scriptable workflows.

Tokens
48.6K
Snippets
155
Records
201
Agent score
77%

What's inside Foundation Lab

  1. Overview of Foundation Models Playgrounds

    main

    The BookPlaygrounds collection provides hands-on Swift playground examples for learning Apple's Foundation Models framework. The learning paths are organized into four progressive modules:

    • GettingStartedWithSessions: Focuses on core session management and basic interactions.
    • GenerationOptionsAndSamplingControl: Covers fine-tuning model behavior and output via generation parameters.
    • BasicToolUse: Demonstrates extending model capabilities using custom tools and function calling.
    • SupportedLanguagesAndInternationalization: Explores multilingual capabilities and internationalization features.

  2. What is DynamicGenerationSchema and when to use it

    main

    Unlike the static @Generable attribute, DynamicGenerationSchema allows you to build schemas at runtime. This is essential for scenarios where the schema structure is not known at compile time, such as:

    • Schema structure is determined by user input.
    • Adapting to changing data formats.
    • Building generic tools that work with various data structures.
    • Creating form builders or configuration-driven extraction systems.

  3. Use GenerationID for stable identity in streaming responses

    main

    The GenerationID type provides a unique identifier that is stable for the duration of a single model response, but may change between different responses.

    When using LanguageModelSession to stream responses, the framework guarantees that a GenerationID will be present and stable. This makes it suitable for use as an identifier in SwiftUI ForEach loops or other collections where the model's output is being updated incrementally (e.g., as text or structured data is being streamed), preventing UI flickering or incorrect state updates that might occur if you used a non-stable property like a name.

    @Generable struct Person: Equatable {
    var id: GenerationID
    var name: String
}

// In a SwiftUI View:
List {
    ForEach(people) { person in
        Text("Name: \(person.name)")
    }
}

  4. Compare adapters in the Foundation Lab macOS app

    main

    For qualitative inspection and interactive timing, use the Adapter Comparison workspace within the Foundation Lab macOS app.

    How it works:

    1. Import a .fmadapter package into the workspace.
    2. The app streams the same prompt through two concurrent sessions: a fresh base-model session and an adapter-model session.
    3. This allows for side-by-side comparison of outputs and latency.

    Note on Latency: Because the two streams run concurrently, the latency numbers provided are intended for diagnostic purposes and should not be used as publishable benchmark results. For controlled benchmarking (warmups, repetitions, randomization, and deterministic grading), use FoundationModelsBench instead.

    Requirements:

    • macOS 26 or later with Apple Intelligence enabled.

  5. How DynamicProfiles and Profiles work together

    main

    The DynamicProfile system allows a LanguageModelSession to switch between different configurations (model, temperature, reasoning level, tools, etc.) dynamically as your app's state changes.

    • DynamicProfile: The top-level coordination layer. It uses a body property to resolve to a single active Profile based on application logic.
    • Profile: A concrete implementation of DynamicProfile that binds DynamicInstructions (content and tools) to specific session-level configuration values (like .temperature or .reasoningLevel).
    • AnyDynamicProfile: A type-erased wrapper used to create instances from specific dynamic profiles.

    Transitions are managed by the body property, which must resolve to a single profile.

    struct PresentationProfile: LanguageModelSession.DynamicProfile {
    var isEditingImage = true
    var isEditingAnimation = false

    var body: some LanguageModelSession.DynamicProfile {
        if isEditingImage {
            // Return an image editing profile
        } else if isEditingAnimation {
            // Return an animation editing profile
        } else {
            // Return a default profile
        }
    }
}

  6. Understand Image Input Probe pass criteria

    main

    A test run is considered a 'pass' only if the following conditions are met:

    1. fm respond exits successfully.
    2. The model returns a non-empty response.
    3. The response includes every term specified via the --expect flag.

    The tool distinguishes between transport status (API success/failure) and semantic status (whether the model's answer was actually correct).

    By default, the tool generates a timestamped results.jsonl report located in /tmp/foundation-lab-image-probe/. If an --output-dir is provided, the report is saved there instead.

  7. Understand the role of FoundationLabCore

    main

    FoundationLabCore is a shared foundational package designed to define the core logic and data structures for the Foundation Lab ecosystem. It acts as a central seam for capabilities, models, and provider abstractions.

    Key Responsibilities:

    • Defining task-oriented capability boundaries.
    • Owning shared request and result models.
    • Owning shared domain models.
    • Defining domain-level errors.
    • Defining provider protocols and shared capability implementations.

    What it is NOT:

    • It does not own delivery layers (UI) or navigation concerns.

  8. Understand the Generable protocol and @Generable macro

    main

    The Generable protocol allows types to be used in contexts involving model-generated content. Conformance is typically provided automatically via the @Generable macro.

    Types conforming to Generable (such as String, Int, Float, and Double) provide access to:

    • generationSchema: A GenerationSchema describing the expected format.
    • promptRepresentation: A Prompt representing the instance as a prompt.
    • instructionsRepresentation: An Instructions object representing the instance as instructions.
    • generatedContent: A GeneratedContent representation of the instance.
    • init(_ content: GeneratedContent): An initializer to create the type from model output.

  9. Use DynamicInstructions for conditional model behavior

    main

    The DynamicInstructions protocol allows you to declaratively assemble instructions and tools for a LanguageModelSession. Unlike static instructions, DynamicInstructions are evaluated before every request, meaning you can use conditional logic (like if/else) to change the model's instructions or available tools based on the current state of your application.

    To implement, conform to DynamicInstructions and provide a body using the @DynamicInstructionsBuilder syntax.

    struct PresentationInstructions: DynamicInstructions {
    // The data source for conditional instructions.
    var isEditingImage = true

    var body: some DynamicInstructions {
        // Static instructions and tools
        Instructions {
            "Help people improve their presentation."
        }
        ListPhotosTool()
        AddPhotoTool()

        // Conditional instructions based on app state
        if isEditingImage {
            ImageEditingInstructions()
        }
    }
}

  10. Use DynamicInstructions for conditional prompting

    main

    The framework provides mechanisms for creating dynamic instructions that can change based on runtime conditions.

    • AnyDynamicInstructions: A type-erased wrapper for any type conforming to DynamicInstructions.
    • ConditionalDynamicInstructions<TrueContent, FalseContent>: Allows you to select between two different sets of instructions based on a condition. It uses a Branch enum to determine which path to take:
      • .trueContent(TrueContent)
      • .falseContent(FalseContent)

  11. Use Dynamic Profiles for reusable model configurations

    main

    Dynamic Profiles allow you to create reusable, structured wrappers around model instructions and behaviors.

    Key components include:

    • DynamicProfile: A protocol for defining model behavior.
    • DynamicProfileModifier: A protocol used to create reusable wrappers that modify existing profiles.
    • ConditionalDynamicProfile: A profile that branches between TrueContent and FalseContent based on conditions.
    • DynamicProfileBuilder: A result builder used to compose profiles.

    You can initialize a session directly with a profile using init(profile:history:).

  12. Manage tool calling requirements with ToolCallingMode

    main

    The ToolCallingMode determines how the model is allowed to use tools during a request. It is a property of GenerationOptions.

    Available Modes

    • .allowed: The model may or may not call tools. This is the default behavior.
    • .required: The model must call one or more tools before it can respond.
      • Warning: If you use .required, you must define an exit condition. Otherwise, the model will loop indefinitely. You can exit by having a tool's call(arguments:) method throw an error or by dynamically changing the mode using a LanguageModelSession.DynamicProfile.
    • .disallowed: The model cannot call any tools and will respond using only its internal knowledge.

    Dynamic Tool Calling Pattern

    To allow a model to call a tool once and then provide a final response, use a DynamicProfile to switch from .required to .allowed after the first tool call.

    extension SessionPropertyValues {
    @SessionProperty
    var toolCallCount: Int = 0
}

struct RecipeDynamicProfile: LanguageModelSession.DynamicProfile {
    @SessionProperty(\.toolCallCount)
    var toolCallCount
    
    var body: some LanguageModelSession.DynamicProfile {
        Profile {
            BreadDatabaseTool()
        }
        // Switch from required to allowed after the first tool call
        .toolCallingMode(toolCallCount < 1 ? .required : .allowed)
        .onToolCall {
            toolCallCount += 1
        }
    }
}