DOCSEARCHbyAlgolia
Submit Docs

LLMFarm_core

repository·main·Indexed 43 days ago

https://github.com/guinmoon/llmfarm_core.swift

A Swift library for running Large Language Models (LLM) on iOS 16+ and macOS 13+. It leverages ggml and llama.cpp to provide high-performance inference with Metal acceleration and various sampling techniques. The package includes tools for LoRA fine-tuning of Llama-based models, merging adapters via the export-lora CLI, and multi-adapter prediction.

Tokens
2.4K
Snippets
7
Records
9
Agent score
55%

What's inside LLMFarm_core

  1. Overview of llmfarm_core features and capabilities

    main

    LLMFarm_core is a Swift library designed for running Large Language Models (LLM) on Apple platforms.

    Supported Platforms:

    • MacOS (13+)
    • iOS (16+)

    Key Capabilities:

    • Hardware Acceleration: Supports Metal (Note: Metal support may not work on Intel-based Macs).
    • Inference & Sampling: Supports various inference types and sampling methods including Temperature (temp, tok-k, top-p), Tail Free Sampling (TFS), Locally Typical Sampling, Mirostat, Greedy, and Grammar-based sampling.
    • Model Management: Includes model setting templates.

  2. Predict using LoRA adapters

    main

    Once a LoRA adapter is trained, you can use it with the ./bin/main executable by passing the base model with -m and the adapter with --lora.

    You can also mix multiple LoRA adapters together in a single prediction session. To control the influence (strength) of each adapter, use the --lora-scaled FN SCALE flag instead of --lora. Scale values do not need to sum to 1, and values greater than 1 can increase influence, though excessively high values may degrade output quality.

    # Predict with a single adapter
./bin/main -m open-llama-3b-v2-q8_0.gguf --lora lora-open-llama-3b-v2-q8_0-shakespeare-LATEST.bin

# Mix multiple adapters with different scales
./bin/main -m open-llama-3b-v2-q8_0.gguf \
  --lora-scaled lora-open-llama-3b-v2-q8_0-shakespeare-LATEST.bin 0.4 \
  --lora-scaled lora-open-llama-3b-v2-q8_0-bible-LATEST.bin 0.8 \
  --lora lora-open-llama-3b-v2-q8_0-yet-another-one-LATEST.bin

  3. Fine-tune a LoRA adapter

    main

    Use the ./bin/finetune command to perform LoRA fine-tuning. Note: Only Llama-based models are supported.

    To manage training progress, use --checkpoint-in and --checkpoint-out. If the input checkpoint does not exist, the process starts with a new randomly initialized adapter. You can use --use-checkpointing to enable gradient checkpointing, which reduces memory requirements by approximately 50% at the cost of increased runtime. If you have sufficient RAM, you can use --no-checkpointing to speed up the process.

    Output files are saved every N iterations based on the --save-every N flag. The filename pattern ITERATION is replaced by the current iteration number, and LATEST always points to the most recent output.

    # Example fine-tuning command
./bin/finetune \
        --model-base open-llama-3b-v2-q8_0.gguf \
        --checkpoint-in  chk-lora-open-llama-3b-v2-q8_0-shakespeare-LATEST.gguf \
        --checkpoint-out chk-lora-open-llama-3b-v2-q8_0-shakespeare-ITERATION.gguf \
        --lora-out lora-open-llama-3b-v2-q8_0-shakespeare-ITERATION.bin \
        --train-data "shakespeare.txt" \
        --save-every 10 \
        --threads 6 --adam-iter 30 --batch 4 --ctx 64 \
        --use-checkpointing

  4. Debug llmfarm_core performance and settings

    main
    When debugging the llmfarm_core package, be aware that the debug version is significantly slower than the release version. To ensure proper debugging behavior, you should comment out the .unsafeFlags(["-Ofast"]) setting in your Package.swift file.

  5. Export a model with LoRA adapters using export-lora

    main

    The export-lora CLI tool allows you to merge LoRA (Low-Rank Adaptation) adapters into a base model and save the result as a new GGUF model file. You can apply multiple adapters in a single command by providing multiple -l or -s flags.

    ./bin/export-lora \
    -m open-llama-3b-v2-q8_0.gguf \
    -o open-llama-3b-v2-q8_0-english2tokipona-chat.gguf \
    -l lora-open-llama-3b-v2-q8_0-english2tokipona-chat-LATEST.bin

  6. Install llmfarm_core via Swift Package Manager

    main

    You can integrate llmfarm_core into your Swift project using Xcode or by manually editing your Package.swift file. This library provides a Swift interface for working with large language models (LLM) based on ggml and llama.cpp.

    dependencies: [
  .package(url: "https://github.com/guinmoon/llmfarm_core.swift")
]

  7. Reference the export-lora CLI options

    main

    The export-lora command accepts the following options to control the model merging process:

    options:
  -h, --help                         show this help message and exit
  -m FNAME, --model-base FNAME       model path from which to load base model (default '')
  -o FNAME, --model-out FNAME        path to save exported model (default '')
  -l FNAME, --lora FNAME             apply LoRA adapter
  -s FNAME S, --lora-scaled FNAME S  apply LoRA adapter with user defined scaling S
  -t N, --threads N                  number of threads to use during computation (default: 4)

  8. Configure LoRA ranks for fine-tuning

    main

    You can customize the LoRA rank for specific model tensors during fine-tuning. The default rank is set by --lora-r N. Note that the rank for all 'norm' tensors should always be set to 1.

    # Reference of rank configuration flags
--lora-r N                 LORA r: default rank. Also specifies resulting scaling together with lora-alpha. (default 4)
--rank-att-norm N          LORA rank for attention norm tensor (default 1)
--rank-ffn-norm N          LORA rank for feed-forward norm tensor (default 1)
--rank-out-norm N          LORA rank for output norm tensor (default 1)
--rank-tok-embd N          LORA rank for token embeddings tensor (default 4)
--rank-out N               LORA rank for output tensor (default 4)
--rank-wq N                LORA rank for wq tensor (default 4)
--rank-wk N                LORA rank for wk tensor (default 4)
--rank-wv N                LORA rank for wv tensor (default 4)
--rank-wo N                LORA rank for wo tensor (default 4)
--rank-w1 N                LORA rank for w1 tensor (default 4)
--rank-w2 N                LORA rank for w2 tensor (default 4)
--rank-w3 N                LORA rank for w3 tensor (default 4)

  9. Reference: Fine-tuning CLI options

    main

    The following flags are used with the ./bin/finetune command to control the training process.

    --model-base FN             Base model file
--checkpoint-in FN          Input checkpoint file
--checkpoint-out FN        Output checkpoint file
--lora-out FN               Output LoRA adapter file (.bin)
--train-data FN             Training data file
--save-every N              Save files every N iterations
--threads N                 Number of threads
--adam-iter N               Adam iterations
--batch N                   Batch size
--ctx N                     Context size
--use-checkpointing         Enable gradient checkpointing
--no-checkpointing          Disable gradient checkpointing
--lora-r N                  Default LoRA rank