DOCSEARCHbyAlgolia
Submit Docs

SwiftWhisper

repository·master·Indexed 21 days ago

https://github.com/exphat/swiftwhisper

A Swift wrapper for whisper.cpp that enables Whisper-based speech-to-text transcription in iOS and macOS applications. It supports asynchronous transcription of 16kHz PCM audio frames, CoreML acceleration via -encoder.mlmodelc files, and progress monitoring through the WhisperDelegate protocol.

Tokens
1.7K
Snippets
5
Records
7
Agent score
24%

What's inside SwiftWhisper

  1. Enable CoreML support

    master
    To use CoreML acceleration, place a CoreML model file with the suffix -encoder.mlmodelc in the same directory as your Whisper model file. For example, if your model is tiny.bin, the CoreML file should be named tiny-encoder.mlmodelc. You must use the Whisper(fromFileURL:) initializer to enable this. You can verify CoreML is active by checking the console output during transcription.

  2. Improve transcription performance during development

    master

    Transcription may be slow in Debug builds because the compiler does not fully optimize the code. To speed up development, you can either:

    1. Configure your scheme to build in the Release configuration.
    2. Use the fast branch of SwiftWhisper, which uses .unsafeFlags(["-O3"]) to force maximum optimization even in non-release builds.
      ...
  dependencies: [
    // Using latest commit hash for `fast` branch:
    .package(url: "https://github.com/exPHAT/SwiftWhisper.git", revision: "deb1cb6a27256c7b01f5d3d2e7dc1dcc330b5d01"),
  ],
  ...

  4. Install SwiftWhisper via Swift Package Manager

    master

    Add SwiftWhisper as a dependency in your Package.swift file. You can target the master branch for the latest features or a specific revision from the fast branch to improve development performance.

    let package = Package(
  ...
  dependencies: [
    // Add the package to your dependencies
    .package(url: "https://github.com/exPHAT/SwiftWhisper.git", branch: "master"),
  ],
  ...
  targets: [
    // Add SwiftWhisper as a dependency on any target you want to use it in
    .target(name: "MyTarget",
            dependencies: [.byName(name: "SwiftWhisper")])
  ]
  ...
)

  5. Transcribe audio with Whisper

    master

    To perform transcription, initialize a Whisper instance with the URL of a model file and call transcribe(audioFrames:) with 16kHz PCM audio frames. This method is asynchronous.

    import SwiftWhisper

let whisper = Whisper(fromFileURL: /* Model file URL */)
let segments = try await whisper.transcribe(audioFrames: /* 16kHz PCM audio frames */)

print("Transcribed audio:", segments.map(\.text).joined())

  6. Convert audio to 16kHz PCM using AudioKit

    master

    SwiftWhisper requires 16kHz PCM audio frames. The following example uses AudioKit to convert an audio file into an array of 16kHz PCM floats.

    import AudioKit

func convertAudioFileToPCMArray(fileURL: URL, completionHandler: @escaping (Result<[Float], Error>) -> Void) {
    var options = FormatConverter.Options()
    options.format = .wav
    options.sampleRate = 16000
    options.bitDepth = 16
    options.channels = 1
    options.isInterleaved = false

    let tempURL = URL(fileURLWithPath: NSTemporaryDirectory()).appendingPathComponent(UUID().uuidString)
    let converter = FormatConverter(inputURL: fileURL, outputURL: tempURL, options: options)
    converter.start { error in
        if let error {
            completionHandler(.failure(error))
            return
        }

        let data = try! Data(contentsOf: tempURL) // Handle error here

        let floats = stride(from: 44, to: data.count, by: 2).map {
            return data[$0..<$0 + 2].withUnsafeBytes {
                let short = Int16(littleEndian: $0.load(as: Int16.self))
                return max(-1.0, min(Float(short) / 32767.0, 1.0))
            }
        }

        try? FileManager.default.removeItem(at: tempURL)

        completionHandler(.success(floats))
    }
}

  7. Implement WhisperDelegate for transcription updates

    master

    You can monitor transcription progress, receive new text segments as they are processed, or handle errors by implementing the WhisperDelegate protocol and assigning it to whisper.delegate.

    protocol WhisperDelegate {
  // Progress updates as a percentage from 0-1
  func whisper(_ aWhisper: Whisper, didUpdateProgress progress: Double)

  // Any time a new segments of text have been transcribed
  func whisper(_ aWhisper: Whisper, didProcessNewSegments segments: [Segment], atIndex index: Int)
  
  // Finished transcribing, includes all transcribed segments of text
  func whisper(_ aWhisper: Whisper, didCompleteWithSegments segments: [Segment])

  // Error with transcription
  func whisper(_ aWhisper: Whisper, didErrorWith error: Error)
}