DOCSEARCHbyAlgolia
Submit Docs

AutoTrain Advanced Documentation

repository·main·Indexed 26 days ago

https://github.com/huggingface/autotrain-advanced

A no-code solution for training and deploying state-of-the-art machine learning models. Supports LLM finetuning (SFT, DPO, ORPO), text classification, regression, Seq2Seq, token classification, and image tasks. Features include a web UI, a CLI for YAML-based configurations, and a programmatic API for project creation. Compatible with local environments and Google Colab.

Tokens
25.5K
Snippets
64
Records
139
Agent score
88%

What's inside AutoTrain Advanced

  2. Supported Sentence Transformer training types

    main

    AutoTrain supports five distinct fine-tuning modes for Sentence Transformers. Choose the mode that matches your dataset structure:

    • pair: Training with two sentences (anchor and positive).
    • pair_class: Training with two sentences (premise and hypothesis) and a target label.
    • pair_score: Training with two sentences (sentence1 and sentence2) and a target score.
    • triplet: Training with three sentences (anchor, positive, and negative).
    • qa: Training with two sentences (query and answer).

  3. Train Image Classification or Regression locally

    main

    To perform local training, create a config.yaml file specifying the task, base_model, backend: local, and your data/parameter configurations.

    Example Image Classification Config:

    task: image_classification
base_model: google/vit-base-patch16-224
project_name: autotrain-cats-vs-dogs-finetuned
log: tensorboard
backend: local

data:
  path: cats_vs_dogs
  train_split: train
  valid_split: null
  column_mapping:
    image_column: image
    target_column: label

params:
  epochs: 2
  batch_size: 4
  lr: 2e-5
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 1
  mixed_precision: fp16

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

    Example Image Regression Config:

    task: image_regression
base_model: microsoft/resnet-50
project_name: autotrain-img-quality-resnet50
log: tensorboard
backend: local

data:
  path: abhishek/img-quality-full
  train_split: train
  valid_split: null
  column_mapping:
    image_column: image
    target_column: target

params:
  epochs: 10
  batch_size: 8
  lr: 2e-3
  optimizer: adamw_torch
  scheduler: cosine
  gradient_accumulation: 1
  mixed_precision: fp16

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true

    Run the training using the CLI:

    autotrain --config config.yaml
    autotrain --config config.yaml

  4. Install AutoTrain Advanced locally

    main

    Install AutoTrain Advanced using pip. It is highly recommended to use a virtual environment (like Conda) to prevent dependency conflicts.

    Important Note: AutoTrain does not install large dependencies like pytorch, torchaudio, or torchvision automatically. You must install these separately. For optimal performance, you may also want to install flash-attn, deepspeed, and xformers.

    $ conda create -n autotrain python=3.10
$ conda activate autotrain
$ pip install autotrain-advanced
$ conda install pytorch torchvision torchaudio pytorch-cuda=12.1 -c pytorch -c nvidia
$ conda install -c "nvidia/label/cuda-12.1.0" cuda-nvcc
$ conda install xformers -c xformers
$ python -m nltk.downloader punkt
$ pip install flash-attn --no-build-isolation # if you want to use flash-attn
$ pip install deepspeed # if you want to use deepspeed

  5. Preview the documentation locally

    main

    To view the documentation in a browser, use the doc-builder preview command. The documentation will be served at http://localhost:5173.

    Important Notes:

    • The preview command only works with existing doc files.
    • If you add a completely new file, you must update _toctree.yml and restart the preview command (stop it with ctrl-c and run it again).
    doc-builder preview autotrain docs/source/

  6. Best practices for LLM fine-tuning

    main

    Memory Optimization

    • Adjust block_size and model_max_length based on available hardware.
    • Enable mixed precision training.
    • Use PEFT (Parameter-Efficient Fine-Tuning) techniques for large models.

    Data Quality

    • Clean and validate training data.
    • Ensure conversation samples are balanced.
    • Use appropriate chat templates.

    Training Tips

    • Start with small learning rates.
    • Monitor training metrics using TensorBoard.
    • Validate model outputs during the training process.

  7. Perform Local LLM Finetuning via CLI

    main

    To perform LLM finetuning locally, create a config.yaml file containing your training parameters and run the autotrain command.

    Key configuration sections include:

    • task: The training task (e.g., llm-orpo).
    • base_model: The model identifier from Hugging Face.
    • data: Specifies the path (Hugging Face dataset or local directory), train_split, chat_template, and column_mapping (mapping dataset columns to text_column, rejected_text_column, and prompt_text_column).
    • params: Training hyperparameters like epochs, batch_size, lr, peft, quantization, and target_modules.
    • hub: Configuration for pushing the model to the Hugging Face Hub, including username, token, and push_to_hub.
    task: llm-orpo
base_model: meta-llama/Meta-Llama-3-8B-Instruct
project_name: autotrain-llama3-8b-orpo
log: tensorboard
backend: local

data:
  path: argilla/distilabel-capybara-dpo-7k-binarized
  train_split: train
  valid_split: null
  chat_template: chatml
  column_mapping:
    text_column: chosen
    rejected_text_column: rejected
    prompt_text_column: prompt

params:
  block_size: 1024
  model_max_length: 8192
  max_prompt_length: 512
  epochs: 3
  batch_size: 2
  lr: 3e-5
  peft: true
  quantization: int4
  target_modules: all-linear
  padding: right
  optimizer: adamw_torch
  scheduler: linear
  gradient_accumulation: 4
  mixed_precision: fp16

hub:
  username: ${HF_USERNAME}
  token: ${HF_TOKEN}
  push_to_hub: true
    $ autotrain --config config.yaml

  9. Prepare data for Seq2Seq tasks

    main

    Seq2Seq tasks (such as machine translation, summarization, or question answering) require a dataset with exactly two columns: text and target. You can provide your data in either CSV or JSONL format.

    CSV Format:

    text,target
"this movie is great","dieser Film ist großartig"
"this movie is bad","dieser Film ist schlecht"

    JSONL Format:

    {"text": "this movie is great", "target": "dieser Film ist großartig"}
{"text": "this movie is bad", "target": "dieser Film ist schlecht"}

  12. Run the AutoTrain API server

    main

    To use the AutoTrain API, install autotrain-advanced and start the application server using the autotrain app command. You can specify the host and port to control where the API is accessible. Once running, the interactive API documentation (Swagger/OpenAPI) is available at the specified host and port under the /docs path.

    $ autotrain app --port 8000 --host 127.0.0.1