Auto-Antislop

FTPO is a preference optimization training algorithm that focuses gradient updates exclusively on the continuation token (the choice between a rejected and a chosen token) rather than the entire preceding context.

Goal: To push probability mass away from the first token of a banned phrase (the rejected token) and toward viable alternatives (the chosen tokens) while keeping the rest of the model's distribution stable.

Training Example Structure:

• Prompt: The shared context up to the point of the ban.
• Rejected Token: The specific token that triggered the ban.
• Chosen Tokens: A set of coherent alternatives sampled using a min-p filter.

Loss Mechanism:

• Preference Term: Encourages the model to rank all chosen tokens higher than the rejected token. Once a chosen token's logit beats the rejected one by a specific margin (ftpo_clip_epsilon_logits), the loss for that pair stops contributing.
• MSE Regularizer: A two-part Mean Squared Error loss in logit space (rather than KL loss) to minimize unnecessary weight movement:
  • mse_target: Applies targeted pressure to the chosen and rejected tokens. Users should typically enable either mse_target_tokenwise or mse_target_aggregate.
  • mse_non_target: Applies loss to the rest of the vocabulary to maintain stability.

Auto-Antislop operates through an iterative pipeline designed to identify and eliminate 'slop' (undesirable linguistic patterns) from model outputs using a combination of generation, analysis, and preference optimization.

Pipeline Flow

1. vLLM Server Management: If manage_vllm is enabled, the script manages a vLLM server. Otherwise, it connects to an existing external vLLM server.
2. Antislop Iteration Loop (num_iterations times):
   • Generation (Iteration 0 - Baseline): Generates text from a source dataset without any active ban lists to establish a baseline.
   • Analysis: Uses slop-forensics to compare n-gram frequencies against human profiles, identify over-represented words, and extract common 'slop phrases'.
   • Ban List Update: Creates or updates n-gram and slop phrase ban lists. This merges findings from analysis with user-supplied extra_ngrams_to_ban, extra_slop_phrases_to_ban, and extra_regex_patterns from the configuration.
   • Generation (Iteration 1+ - Anti-Slop): Re-generates text using the accumulated ban lists. When a ban occurs, the system captures preference pairs (rejected token vs. coherent alternatives) to build a training dataset.
3. FTPO Dataset Creation: Creates preference pairs consisting of a prompt (context up to the ban), a rejected token (the first token of the banned sequence), and chosen tokens (coherent alternatives selected via a min-p filter).
4. FTPO Finetuning: If finetune_enabled is true, the pipeline performs FTPO finetuning using the generated preference dataset. This supports LoRA, 4-bit quantization, and both unsloth and transformers/trl training paths.

To perform a complete unslop process (generating a baseline, identifying slop, creating a preference dataset, and fine-tuning) using the provided example for gemma-3-4b-it, follow these steps to set up a virtual environment and execute the pipeline:

1. Clone the repository with submodules.
2. Create and activate a Conda environment.
3. Install specific versions of torch and the project requirements.
4. Install vllm.
5. Run the main script using the provided configuration.

Note: This sequence is designed for a quick test of the pipeline's functionality.

# creates a venv, installs auto-antislop and runs an end-to-end unslop of gemma-3-4b-it
git clone https://github.com/sam-paech/auto-antislop.git --recurse
conda create -n antislop python=3.11 -y
conda activate antislop
cd auto-antislop
pip install "torch==2.8.*"
pip install -r requirements.txt
pip install vllm
python main.py -c configs/gemma-3-4b-it.yaml

Prerequisites

• Python 3.9+
• NVIDIA GPU with CUDA installed
• Git

Installation Steps

1. Clone the Repository (ensure submodules are included):

   git clone --recurse-submodules https://github.com/your-username/auto-antislop.git
   cd auto-antislop

   If you already cloned without submodules, run: git submodule update --init --recursive

2. Install Dependencies: It is highly recommended to use a virtual environment.

   pip install -r requirements.txt

   Note: vllm and torch are commented out in requirements.txt to allow you to install versions compatible with your specific CUDA setup. If using the --manage-vllm feature, install vllm manually (e.g., pip install vllm).

3. NLTK Data: The script attempts to download punkt, punkt_tab, and stopwords automatically. If this fails, download them manually in Python:

   import nltk
   nltk.download('punkt')
   nltk.download('punkt_tab')
   nltk.download('stopwords')

You can fine-tune the FTPO training process by setting specific hyper-parameters in your configuration file. If these are omitted or set to null, the system falls back to FTPOTrainer defaults.

Available Hyper-parameters

# Example FTPO configuration snippet
ftpo_beta: 0.1
ftpo_lambda_mse_target_tokenwise: 0.05
ftpo_tau_mse_target_tokenwise: 0.5
ftpo_lambda_mse: 0.4
ftpo_clip_epsilon_logits: 2

Configuration is managed through a YAML file (examples are in configs/). Key configuration groups include:

VLLM & Antislop Generation

• model_id: HuggingFace ID or local directory of the model to unslop.
• generation_threads: Number of parallel threads for generation (try 30).
• generation_max_prompts: Number of prompts to generate (e.g., 50 for testing, 1000-2000 for production).
• generation_hf_dataset_name: HuggingFace dataset (ShareGPT format) used to source prompts.
• extra_slop_phrases_to_ban: Custom list of strings to be trained out of the model.

Fine-Tuning

• finetune_use_unsloth: Use unsloth (lower VRAM) or transformers/trl.
• finetune_mode: Options are ftpo (recommended for gentler training), dpo, or dpo_final_token.
• finetune_early_stopping_wins: Threshold (e.g., 0.8-0.9) to stop training when 'chosen' tokens are preferred over 'rejected' tokens.
• finetune_lora_r: LoRA rank. FTPO works best with high ranks (128-256).
• finetune_target_modules: Modules to target (e.g., ["lm_head"] for minimal intervention).
• finetune_freeze_early_layers: Boolean to freeze all layers except the last n layers (finetune_n_layers_unfrozen).
• finetune_learning_rate: Manual learning rate.
• finetune_auto_learning_rate: Boolean to use an automatic learning rate based on dataset/batch size.
• finetune_max_train_examples: Number of training examples (suggested 8000-12000).