SwiftOpenAI

SwiftOpenAI is compatible with Apple platforms and Linux:

• Apple Platforms: iOS 15+, macOS 13+, and watchOS 9+.
• Linux: Uses AsyncHTTPClient to bypass URLSession bugs in Apple's Foundation framework. It is compatible with the Vapor server framework.

It also supports various OpenAI-compatible providers such as Azure OpenAI, Anthropic, Gemini, Ollama, Groq, xAI, OpenRouter, Tuning Engines, DeepSeek, and AIProxy. You can use OpenAIServiceFactory to provide custom URLs for these providers.

SwiftOpenAI provides streamlined interaction with the following OpenAI API categories:

• Audio: Transcriptions, Translations, Speech, and the Realtime API (low-latency bidirectional voice).
• Chat: Function Calling, Structured Outputs, and Vision.
• Response: Streaming Responses.
• Embeddings
• Fine-tuning
• Batch
• Files
• Images
• Models
• Moderations

BETA Features:

• Assistants: Assistants File Object, Threads, Messages (including Message File Object), Runs (including Run Step object/details), Assistants Streaming (Message Delta and Run Step Delta), and Vector Stores (File and File Batch).

A MessageObject contains content that can be either text or images. When the assistant uses tools like retrieval or code_interpreter, the text content may include annotations that provide context for citations or file paths.

Content Types

• text: Contains a TextContent object with the string value and an array of annotations.
• imageFile: References an image via a fileID.

Annotations

• fileCitation: Used when the assistant uses the retrieval tool. It includes the fileID and the specific quote from the file.
• filePath: Used when the assistant uses the code_interpreter tool. It includes the fileID of the generated file.

The Realtime API enables bidirectional voice conversations using WebSockets.

Platform Requirements:

• iOS 15+, macOS 13+, watchOS 9+.
• Requires AVFoundation (not available on Linux).

Permissions Required:

• Add NSMicrophoneUsageDescription to your Info.plist.
• On macOS: Enable sandbox entitlements for microphone access and outgoing network connections.

To enable function calling, define a Tool containing a ChatFunction.

ChatFunction properties:

• name: The function name (a-z, A-Z, 0-9, underscores, or dashes; max 64 chars).
• description: A description of what the function does.
• parameters: A JSONSchema object describing the arguments.
• strict: A boolean to enable strict schema adherence (Structured Outputs).

Note: FunctionCall is deprecated in favor of ToolChoice and tool_calls.

let myFunction = ChatFunction(
    name: "get_weather",
    strict: true,
    description: "Get the current weather",
    parameters: myJsonSchema
)
let tool = Tool(type: "function", function: myFunction)

To ensure correct dimension constraints for DALL-E models, use the Dalle enum. This helps avoid API errors by mapping model types to their supported sizes.

• dalle2: Supports small (256x256), medium (512x512), and large (1024x1024).
• dalle3: Supports largeSquare (1024x1024), landscape (1792x1024), and portrait (1024x1792).

// Using the Dalle enum to get correct model and size strings
let dalleConfig = Dalle.dalle3(.landscape)
let modelName = dalleConfig.model // "dall-e-3"
let sizeString = dalleConfig.size // "1792x1024"

The Response API accepts inputs via the InputType enum, which supports two modes:

1. .string(String): Simple text input.
2. .array([InputItem]): An array of items for complex or multimodal conversations.

InputItem can be:

• .message(InputMessage): User, assistant, or system messages.
• .functionToolCall(FunctionToolCall): Function calls.
• .functionToolCallOutput(FunctionToolCallOutput): Function outputs.

InputMessage requires a role (user, assistant, or system) and content (either .text(String) or .array([ContentItem]) for multimodal data).

To use OpenAI's Structured Outputs, you must follow these requirements:

• All fields must be required: Every field or function parameter in your schema must be specified as required. To emulate an optional parameter, use a union type with null.
• Set additionalProperties: false: This must always be set in objects to opt into Structured Outputs.
• Nesting limits: Schemas are limited to a maximum of 100 object properties total and up to 5 levels of nesting.
• Key ordering: Outputs will follow the order of keys defined in your schema.
• Recursive schemas: These are supported.

The current version of SwiftOpenAI supports Assistants Beta 2.

If your project requires support for Assistants V1, you must use one of the following:

• The jroch-supported-branch-for-assistants-v1 branch.
• The v2.3 release.

SwiftOpenAI supports Assistants API streaming. By passing "stream": true to endpoints like createRun, createThreadAndRun, or submitToolOutputs, you receive a Server-Sent events (SSE) stream. This allows you to react to real-time changes using MessageDeltaObject (for message content changes) and RunStepDeltaObject (for run step changes).

// Note: To use streaming, ensure the 'stream' parameter is set to true in your request.
// You can then handle incoming events such as MessageDeltaObject and RunStepDeltaObject.

Turn detection (Voice Activity Detection) determines when a user has finished speaking. You can choose between two modes:

1. Server-based VAD (.serverVAD): Uses server-side logic with customizable timing parameters:

   • prefixPaddingMs: Default 300
   • silenceDurationMs: Default 500
   • threshold: Default 0.5

2. Semantic VAD (.semanticVAD): Uses semantic analysis with an Eagerness level (.low, .medium, .high).

// Server-based VAD
let serverVAD = TurnDetection.serverVAD(prefixPaddingMs: 300, silenceDurationMs: 500, threshold: 0.5)

// Semantic VAD
let semanticVAD = TurnDetection.semanticVAD(eagerness: .medium)