Lilac Documentation

A concept in Lilac is an algorithmic representation of a natural language idea (e.g., "positive sentiment" or "toxicity"). It allows the AI to generalize from specific examples to identify similar patterns in unseen text.

Technically, a concept is defined as a collection of:

• Positive examples: Text segments that are related to the concept.
• Negative examples: Text segments that are unrelated to the concept (either the opposite or simply irrelevant).

Once a concept is established, Lilac uses it to score text or rank documents within a dataset based on their relevance to that concept.

Lilac Garden is an accelerated computation platform designed to handle large-scale text clustering. It provides a single API that accepts a list of documents and returns human-readable cluster names and categories.

Unlike traditional clustering methods that require manual embedding management and intensive compute pipelines, Lilac Garden uses:

• Long context embeddings
• Massively parallel GPU compute
• Sophisticated LLMs to generate concise, descriptive titles for each cluster.

This service is designed to speed up data curation by allowing users to:

• Identify and remove problematic clusters (e.g., jailbreaks or NSFW content).
• Sub-sample large clusters to reduce dataset size.
• Create task-specific datasets.
• Analyze user-LLM interaction logs to identify hazardous engagement patterns.

A concept is a way to define a specific semantic pattern or category within your data. It is created by seeding it with a set of positive and negative examples, which are then used to tune the concept against a data source. Concepts can be created via the Lilac UI or through the Python API, and these methods are interchangeable.

To make a concept effective, you must provide both:

• Positive examples: Text that represents the concept.
• Negative examples: Text that is either the opposite of the concept or unrelated to it. Negative examples are critical because they allow Lilac to infer the boundaries of what is and is not related to the concept.

By default, select_rows returns a flat dictionary where every path is a top-level key (e.g., 'text.pii.emails').

If you set combine_columns=True, Lilac will return a deeply nested dictionary structure where enrichments and signals live under their original document paths. This is useful for maintaining the relationship between a source field and its derived metadata.

# Returns nested structure: {'text': {'__value__': '...', 'pii': {...}}, 'label': '...'}
rows = dataset.select_rows(
  columns=['text', 'label', ('text', 'pii', 'emails', '*')],
  filters=[(('text', 'pii', 'emails', '*'), 'exists')],
  combine_columns=True,
  limit=1
)

Signals are used to extract metadata from data. They can only be created in Python and are used to enrich datasets with new information. There are two primary base classes for creating signals:

• ll.TextSignal: Takes text as input and returns metadata.
• ll.TextEmbeddingSignal: Takes embeddings as input and returns metadata (this is the base class for ConceptSignals).

Once defined, signals can be used to compute values over individual lists of data or applied to an entire Lilac dataset. After registration, they become available in the Lilac web UI for exploration and previewing.

import lilac as ll

# Example of the two base classes
class MyTextSignal(ll.TextSignal):
    pass

class MyEmbeddingSignal(ll.TextEmbeddingSignal):
    pass

Lilac Signals are used to enrich unstructured data fields with structured metadata. This enrichment enables users to compute statistics, identify problematic data slices, and measure changes in dataset composition over time.

Common examples of signals include:

• Near-duplicate detection: Identifying similar text entries.
• PII Detection: Finding sensitive information like emails, phone numbers, IP addresses, and secrets.
• Text Statistics: Computing metrics like readability, character counts, or language detection.

After a signal has been computed on a field, the results are stored as sub-fields of the original field. When using select_rows, you can retrieve these values by passing a tuple containing the original field name and the signal name: (original_field, signal_name).

For example, if you applied LangDetectionSignal to a field named text, the resulting signal field is accessed as text.lang_detection in the output dictionary.

# Example of selecting the original field and its computed signal result
results = dataset.select_rows([('text', 'lang_detection'), 'text'], limit=1)
print(list(results))

A concept is a collection of positive and negative examples used to identify related text. You can apply a concept to a single document or a dataset. To use a concept, you must specify an Embedding.

Embeddings fall into two categories:

• On-device: e.g., gte, sbert. These run locally.
• API-based: e.g., openai, cohere. These require an API key and make network requests.

The quality of the concept scoring is dependent on the chosen embedding.

For datasets larger than 10,000 rows, or for significantly faster results, use Lilac Garden. Lilac Garden is a remote computation service that handles compute-heavy tasks like clustering, perplexity scoring, and embedding computation.

Key Benefits:

• Speed: A 10,000 point dataset clusters in less than a minute (roughly 100x faster than local computation).
• Scale: Capable of clustering millions of data points (e.g., 4 million points in about an hour).
• Ease of Use: Removes the need to manage local GPU dependencies or monitor long-running local pipelines.

Fields in a Lilac dataset are identified by a path, which represents the hierarchy of the data (including original fields and enriched signal data). Paths can be expressed in two ways:

1. As a tuple: ('text', 'pii', 'emails')
2. As a period-separated string: 'text.pii.emails'

To discover all available leaf paths in your dataset schema, use the leafs property of the data_schema object. This returns a dictionary mapping paths to Field objects.

# Get the dictionary keys from the leafs to see all available paths
leaf_paths = manifest.data_schema.leafs.keys()
print(leaf_paths)

A Lilac Concept is a customizable, AI-powered embedding-based classifier. Unlike static heuristics, Concepts allow you to define what 'good' or 'bad' data looks like for your specific application.

Key features of Concepts:

• Refinement: You can create and refine them directly through the Lilac UI.
• Real-time updates: Concepts can be updated in real-time based on user feedback.
• Application-specific or General: They can be highly specialized (e.g., detecting specific legal clauses) or broadly applicable (e.g., detecting profanity or source code).