GLiClass: Generalist and Lightweight Model for Text Classification

repository·main·Indexed 42 days ago

https://github.com/knowledgator/gliclass

An efficient, zero-shot sequence classification model designed for high-speed performance using a single forward pass. GLiClass supports multi-label classification, hierarchical labels, few-shot examples, and Retrieval-Augmented Classification (RAC). It features specialized pipelines for long documents via automatic chunking and streaming data with session management, and can be deployed in production using Ray Serve.

Tokens: 17.7K
Snippets: 50
Records: 59
Agent score: 58%

What's inside GLiClass

Use hierarchical labels for structured classification

main

GLiClass supports hierarchical label structures using dot notation. You can pass a dictionary where keys are parent labels and values are lists of sub-labels.

If you set return_hierarchical=True, the output will be a nested dictionary matching your input structure instead of a flat list of dot-notated strings.

hierarchical_labels = {
    "sentiment": ["positive", "negative", "neutral"],
    "topic": ["product", "service", "shipping"]
}

text = "The product quality is amazing but delivery was slow"
# Returns flat dot-notation labels: 'sentiment.positive', etc.
results = pipeline(text, hierarchical_labels, threshold=0.5)[0]

# Returns nested dictionary structure
results_nested = pipeline(text, hierarchical_labels, return_hierarchical=True)[0]

How session lifecycle and management works
main
Sessions are managed via session_ids. The set of session_ids provided in a single call represents the complete active set of sessions.
- Creation: A new session is created when a new session_id is provided.
- Extension: An existing session is updated when its session_id is included in a subsequent call.
- Deletion: A session is automatically deleted if its session_id is absent from the next call.
- Retention without text: To keep a session active without adding new text (and without deleting it), include it with an empty string "" in the texts list.
Manual Management Methods:
- pipeline.delete_session(*session_ids): Removes specific sessions.
- pipeline.clear_sessions(): Removes all active sessions.
- pipeline([], [], session_ids=[]): An empty call also clears all sessions.
- pipeline.active_sessions: Returns a list of currently active IDs in insertion order.
```
# Session 'a' is deleted; 'b' is extended; 'c' is created
pipeline([chunk_a, chunk_b], labels, session_ids=["a", "b"])
pipeline([next_b, chunk_c], labels, session_ids=["b", "c"])

assert pipeline.active_sessions == ["b", "c"]

# Retain session 'a' without adding text
pipeline(
    texts=["", next_chunk_b],
    labels=labels,
    session_ids=["a", "b"],
)
```
https://github.com/knowledgator/gliclass

Streaming Classification with Session Management

main

GLiClass supports incremental, multi-session text classification using a decoder-KV model. It maintains a persistent KV cache per session to avoid re-encoding the full document on every update.

Installation:

pip install gliclass[streaming]

Usage: Use StreamingPipeline and provide a SessionInput containing a session_id. You can control when classification occurs using a strategy (e.g., EveryNTokensStrategy).

from gliclass.streaming import StreamingPipeline, SessionInput, EveryNTokensStrategy

pipeline = StreamingPipeline(model, tokenizer, device="cuda", max_cache_len=1024)
strategy = EveryNTokensStrategy(n=50)

for chunk in text_chunks:
    outputs = pipeline([SessionInput(
        session_id="doc_001",
        text=chunk,
        labels=["science", "politics", "finance"],
        strategy=strategy,
        classification_type="multi-label",
    )])
    if outputs[0].triggered:
        print(outputs[0].predictions)

Retrieval-Augmented Classification (RAC)

main

When using models trained for Retrieval-Augmented Classification (e.g., gliclass-base-v2.0-rac-init), you can provide rac_examples to improve accuracy. An example is a dictionary containing text, all_labels (the full set of possible labels), and true_labels (the labels present in that specific text).

example = {
    "text": "A new machine learning platform automates complex data workflows but faces integration issues.",
    "all_labels": ["AI", "automation", "data_analysis", "usability", "integration"],
    "true_labels": ["AI", "integration", "automation"]
}

text = "The new AI-powered tool streamlines data analysis but has limited integration capabilities."
labels = ["AI", "automation", "data_analysis", "usability", "integration"]

results = pipeline(text, labels, threshold=0.1, rac_examples=[example])[0]

Enable Flash Attention backends
main
GLiClass supports optional flash attention backends for faster inference. These are enabled automatically if the packages are installed and the correct environment variables are set.
DeBERTa v2:
1. Install: pip install flashdeberta
2. Enable: export USE_FLASHDEBERTA=1
T5 / mT5:
1. Install: pip install turbot5
2. Enable: export TURBOT5_ATTN_TYPE=triton-basic
https://github.com/knowledgator/gliclass/blob/main/README.md
Install the streaming classification package
main
To use the streaming capabilities of GLiClass, install the package with the [streaming] extra:
```
pip install gliclass[streaming]
```
https://github.com/knowledgator/gliclass

Improve accuracy with Few-Shot examples

main

You can provide in-context examples to the pipeline to improve classification accuracy. Each example should be a dictionary containing the text and the corresponding labels.

examples = [
    {
        "text": "Love this item, great quality!",
        "labels": ["positive", "product"]
    },
    {
        "text": "Customer support was unhelpful",
        "labels": ["negative", "service"]
    }
]

text = "Fast delivery and the item works perfectly!"
labels = ["positive", "negative", "product", "service", "shipping"]

results = pipeline(text, labels, examples=examples, threshold=0.5)[0]

Prepare training data for GLiClass
main
To train or fine-tune GLiClass, prepare your data as a list of JSON objects. Each object should contain the text, all_labels (the full label set), and true_labels (the ground truth). true_labels can be a list of labels or a dictionary mapping labels to confidence scores.
```
[
  {"text": "Sample text.", "all_labels": ["sports", "science", "business"], "true_labels": ["sports"]},
  {"text": "Sample text.", "all_labels": ["sports", "science"], "true_labels": {"sports": 0.9}}
]
```
https://github.com/knowledgator/gliclass/blob/main/README.md

Use Task Description Prompts

main

You can guide the classification task by providing a custom prompt. This can be a single string for a single text, or a list of strings for batch processing.

# Single text prompt
results = pipeline(
    text, 
    labels, 
    prompt="Classify the sentiment of this product review:", 
    threshold=0.5
)[0]

# Batch processing with per-text prompts
texts = ["Review about electronics", "Review about clothing"]
prompts = [
    "Analyze this electronics review:",
    "Analyze this clothing review:"
]
results = pipeline(texts, labels, prompt=prompts)

Install GLiClass

main

You can install GLiClass via pip or directly from the GitHub repository.

Via pip:

pip install gliclass

From source:

git clone https://github.com/Knowledgator/GLiClass
cd GLiClass

python -m venv venv
source venv/bin/activate  # Windows: venv\Scripts\activate

pip install -r requirements.txt
pip install .

Configure CPU offloading for large caches
main
To manage memory when working with many active sessions or large caches, you can offload the KV caches to the CPU. This is particularly useful when the total cache size exceeds GPU memory.
Set offload_to_cpu=True in the constructor. For optimal performance with CUDA, also set use_pinned_memory=True to enable non-blocking transfers.
```
pipeline = StreamingZeroShotClassificationPipeline(
    model,
    tokenizer,
    device="cuda",
    offload_to_cpu=True,
    use_pinned_memory=True,
    score_on_cpu=True,
)
```
https://github.com/knowledgator/gliclass

Deploy GLiClass with Ray Serve

main

GLiClass can be served in production using Ray Serve.

Installation:

pip install gliclass[serve]

Running the server:

# Default model
python -m gliclass.serve

# Specify model and port
python -m gliclass.serve --model knowledgator/gliclass-edge-v3.0 --port 8000

# Using a config file
python -m gliclass.serve --config serve_configs/serve_config.yaml

Using the Python Client:

from gliclass.serve import GLiClassClient

client = GLiClassClient(url="http://localhost:8000/gliclass")
result = client.classify(
    text="This is a great product!",
    labels=["positive", "negative", "neutral"],
    threshold=0.3,
)

Using the HTTP API:

curl -X POST http://localhost:8000/gliclass \
  -H "Content-Type: application/json" \
  -d '{
    "texts": "This is a great product!",
    "labels": ["positive", "negative", "neutral"],
    "threshold": 0.3
  }'

from gliclass.serve import GLiClassClient

client = GLiClassClient(url="http://localhost:8000/gliclass")

result = client.classify(
    text="This is a great product!",
    labels=["positive", "negative", "neutral"],
    threshold=0.3,
)