Adaptive Classifier

repository·main·Indexed 20 days ago

https://github.com/codelion/adaptive-classifier

A PyTorch-based library for dynamic text classification supporting continuous learning, dynamic class addition, and strategic defense against adversarial attacks. It features AdaptiveClassifier for single-label tasks and MultiLabelAdaptiveClassifier for multi-category tasks with automatic threshold adaptation. The library integrates with HuggingFace transformers and the HuggingFace Hub, and utilizes ONNX Runtime for 2-4x faster CPU inference.

Tokens
16.5K
Snippets
55
Records
65
Agent score
69%

What's inside adaptive-classifier

  1. Handle order dependency in online learning

    main

    When using add_examples to perform true online learning (adding examples incrementally), the order of addition can affect predictions because the underlying neural network learns incrementally.

    To achieve strict order independence, you can switch from the default hybrid approach to a Prototype-Only configuration. This makes predictions based solely on similarity to class prototypes (mean embeddings) rather than the neural network component.

    # Scenario 1
    classifier.add_examples(["fish example"], ["aquatic"])
    classifier.add_examples(["bird example"], ["aerial"])
    
    # Scenario 2
    classifier.add_examples(["bird example"], ["aerial"])
    classifier.add_examples(["fish example"], ["aquatic"])
    # These may produce slightly different models due to incremental training.
  2. Understand Multi-Label Adaptive Thresholding

    main

    To prevent the common issue of "No labels met the threshold criteria" in large label sets, the MultiLabelAdaptiveClassifier automatically lowers the threshold as the number of possible labels increases:

    Number of LabelsThresholdBenefit
    2-4 labels0.5 (default)Standard precision
    5-9 labels0.4 (20% lower)Balanced recall
    10-19 labels0.3 (40% lower)Better coverage
    20-29 labels0.2 (60% lower)Prevents empty results
    30+ labels0.1 (80% lower)Ensures predictions
  3. How the Adaptive Classifier architecture works

    main

    The Adaptive Classifier is composed of four integrated components:

    1. Transformer Embeddings: Provides high-quality text representations using SOTA language models.
    2. Prototype Memory: Stores class prototypes to enable rapid adaptation to new examples.
    3. Adaptive Neural Layer: Learns and refines decision boundaries through continuous training.
    4. Strategic Classification: A defense mechanism that models potential adversarial behavior using cost functions and provides multiple prediction modes (dual, strategic, and robust) to ensure stability against manipulation.
  4. Add a new enterprise classifier to the test suite

    main

    To include a new classifier in the integration testing lifecycle, follow these steps:

    1. Update Metrics: Add the new classifier and its expected metrics to the CLASSIFIER_METRICS dictionary.
    2. Add Test Data: Add domain-specific test sentences to the TEST_SENTENCES list.
    3. HuggingFace Deployment: Ensure the model is published to HuggingFace Hub using the naming convention adaptive-classifier/{name}.
  5. Install Adaptive Classifier

    main

    You can install the library via pip. The package includes ONNX Runtime by default, which provides 2-4x faster CPU inference out-of-the-box.

    Quick Install

    pip install adaptive-classifier

    Development Setup

    To install for development purposes, clone the repository and install in editable mode:

    git clone https://github.com/codelion/adaptive-classifier.git
    cd adaptive-classifier
    pip install -e .
  6. Use pre-trained enterprise classifiers

    main

    You can load specialized, pre-trained models using AdaptiveClassifier.from_pretrained(). These are optimized for specific enterprise tasks:

    • Hallucination Detection: adaptive-classifier/llm-hallucination-detector
    • LLM Routing: adaptive-classifier/llm-router
    • Config Optimization: adaptive-classifier/llm-config-optimizer
    • Content Moderation: adaptive-classifier/content-moderation
    • Business Sentiment: adaptive-classifier/business-sentiment
    • PII Detection: adaptive-classifier/pii-detection
    • Fraud Detection: adaptive-classifier/fraud-detection
    • Email Priority: adaptive-classifier/email-priority
    • Compliance Classification: adaptive-classifier/compliance-classification
    # Example: Hallucination Detection
    detector = AdaptiveClassifier.from_pretrained("adaptive-classifier/llm-hallucination-detector")
    context = "France is in Western Europe. Capital: Paris. Population: ~67 million."
    response = "Paris is the capital. Population is 70 million."
    
    prediction = detector.predict(f"Context: {context}\nAnswer: {response}")
    # Returns: [('HALLUCINATED', 0.72), ('NOT_HALLUCINATED', 0.28)]
  7. Optimize Inference with ONNX

    main

    Adaptive Classifier uses ONNX Runtime for high-performance CPU inference.

    Automatic Behavior

    • On CPU: Automatically uses ONNX (2-4x faster than PyTorch).
    • On GPU: Automatically uses PyTorch for maximum performance.

    Loading Strategies

    When loading a saved model, you can control the ONNX behavior:

    • Quantized (Default): Uses INT8 quantization. 4x smaller and significantly faster on x86/ARM.
    • Unquantized: Use prefer_quantized=False for maximum accuracy.
    • Disable ONNX: Use use_onnx=False to force PyTorch.

    Saving Options

    • Include ONNX: Default behavior includes both quantized and unquantized versions.
    • Exclude ONNX: Use include_onnx=False in .save() to skip ONNX export.
    # Save with ONNX export
    classifier.save("./model")
    
    # Load automatically uses quantized ONNX on CPU (fastest, 4x smaller)
    fast_classifier = AdaptiveClassifier.load("./model")
    
    # Choose unquantized ONNX for maximum accuracy
    accurate_classifier = AdaptiveClassifier.load("./model", prefer_quantized=False)
    
    # Force PyTorch (no ONNX)
    pytorch_classifier = AdaptiveClassifier.load("./model", use_onnx=False)
    
    # Opt-out of ONNX export when saving
    classifier.save("./model", include_onnx=False)
  8. Run Enterprise Classifier Integration Tests

    main

    The integration test suite (tests/test_enterprise_classifiers_integration.py) verifies that the 17 enterprise classifiers hosted on Hugging Face Hub maintain expected performance, model loading capabilities, and prediction stability. Use these tests to ensure code changes do not break published models.

    Test Coverage

    • Model Loading: Verifies models can be loaded from HuggingFace Hub.
    • Prediction Functionality: Ensures models make valid predictions.
    • k-Parameter Consistency: Regression test ensuring k=1 and k=2 produce consistent results.
    • Prediction Stability: Checks if repeated predictions are consistent.
    • Performance: Ensures inference completes within 2 seconds.
    • Class Coverage: Verifies models recognize all expected classes.
    • Health Check: Overall ecosystem health assessment.
    # Run all integration tests
    pytest tests/test_enterprise_classifiers_integration.py -v
  9. Add new classes and examples dynamically

    main

    You can expand the classifier's knowledge without full retraining by using add_examples. This allows for both adding entirely new categories and refining existing ones through continuous learning.

    # Add a completely new class
    new_texts = [
        "Error code 404 appeared",
        "System crashed after update"
    ]
    new_labels = ["technical"] * 2
    
    classifier.add_examples(new_texts, new_labels)
    
    # Add more examples to existing classes
    more_examples = [
        "Best purchase ever!",
        "Highly recommend this"
    ]
    more_labels = ["positive"] * 2
    
    classifier.add_examples(more_examples, more_labels)
  10. Save and Load Models

    main

    You can persist your trained classifiers locally or interact with the HuggingFace Hub.

    Local Storage

    • Save: classifier.save("./path")
    • Load: AdaptiveClassifier.load("./path")

    HuggingFace Hub

    • Push: classifier.push_to_hub("repo_id")
    • Load: AdaptiveClassifier.from_pretrained("repo_id")
    # Save locally
    classifier.save("./my_classifier")
    loaded_classifier = AdaptiveClassifier.load("./my_classifier")
    
    # 🤗 HuggingFace Hub Integration
    classifier.push_to_hub("adaptive-classifier/my-model")
    hub_classifier = AdaptiveClassifier.from_pretrained("adaptive-classifier/my-model")
  11. Enable Strategic Classification for anti-gaming defense

    main

    To defend against adversarial inputs (users trying to manipulate the classifier), you can enable strategic_mode via the config dictionary when initializing AdaptiveClassifier. This mode uses game-theoretic principles to model potential manipulation.

    # Configuration for strategic mode
    config = {
        'enable_strategic_mode': True,
        'cost_function_type': 'linear',
        'cost_coefficients': {
            'sentiment_words': 0.5,    # Cost to change sentiment-bearing words
            'length_change': 0.1,      # Cost to modify text length
            'word_substitution': 0.3   # Cost to substitute words
        },
        'strategic_blend_regular_weight': 0.6,   # Weight for regular predictions
        'strategic_blend_strategic_weight': 0.4  # Weight for strategic predictions
    }
    
    classifier = AdaptiveClassifier("bert-base-uncased", config=config)
    classifier.add_examples(texts, labels)
    
    text = "This product has amazing quality features!"
    
    # Dual prediction (automatic blend of regular + strategic)
    predictions = classifier.predict(text)
    
    # Pure strategic prediction (simulates adversarial manipulation)
    strategic_preds = classifier.predict_strategic(text)
    
    # Robust prediction (assumes input may already be manipulated)
    robust_preds = classifier.predict_robust(text)
  12. Basic Text Classification with AdaptiveClassifier

    main

    Use AdaptiveClassifier to perform dynamic text classification. You can initialize it with any HuggingFace transformer model, add training examples at runtime to enable continuous learning, and then make predictions.

    Workflow:

    1. Initialize with a HuggingFace model name.
    2. Use .add_examples(texts, labels) to provide training data.
    3. Use .predict(text) to get class probabilities.
    from adaptive_classifier import AdaptiveClassifier
    
    # Initialize with any HuggingFace model
    classifier = AdaptiveClassifier("bert-base-uncased")
    
    # Add training examples
    texts = ["The product works great!", "Terrible experience", "Neutral about this purchase"]
    labels = ["positive", "negative", "neutral"]
    classifier.add_examples(texts, labels)
    
    # Make predictions
    predictions = classifier.predict("This is amazing!")
    print(predictions)  
    # Output: [('positive', 0.85), ('neutral', 0.12), ('negative', 0.03)]