LightlyStudio

The LightlyStudio backend is a Python package located in lightly_studio/src/lightly_studio. It functions as a local application, a FastAPI server, and a web UI host.

Core Technology Stack

• FastAPI: Handles the HTTP API and application lifecycle.
• Pydantic: Manages request/response models, validation, and OpenAPI schema generation.
• SQLModel: Provides typed database access and models on top of SQLAlchemy.
• Uvicorn: The ASGI server used to run the backend.
• Database Support: Supports DuckDB (default) and PostgreSQL (alternative).

Layered Architecture and Request Flow

Requests typically follow a specific flow to maintain separation of concerns: FastAPI route $\rightarrow$ optional service $\rightarrow$ resolver(s) $\rightarrow$ SQLModel / database.

• Routes (api/): Translate HTTP input into typed models, wire dependencies, and map failures to HTTP responses. Note: Do not raise HTTPException directly in routes; raise specific exceptions and let the API layer handle the conversion.
• Services (services/): An orchestration layer for workflows that span multiple resolvers or require branching business logic. Use these for cross-entity operations.
• Resolvers (resolvers/): The layer responsible for database-facing query and mutation functions. This is where most persistence logic resides.
• Models (models/): Contains SQLModel tables and Pydantic/SQLModel request/view models.

Typicality is a per-sample score derived from embeddings. Samples close to many others receive a high score (typical); outliers receive a low score.

To find outliers, compute typicality metadata and then use metadata_weighting with a negative strength to prefer low-typicality samples. To find a diverse but representative subset, combine typicality with diversity sampling.

import lightly_studio as ls

dataset = ls.ImageDataset.load_or_create()
dataset.add_images_from_path(path="/path/to/image_dataset")

# Compute and store typicality scores as metadata.
dataset.compute_typicality_metadata(metadata_name="typicality")

# Sample the 5 most typical items.
dataset.query().sampling().metadata_weighting(
    n_samples_to_select=5,
    sampling_result_tag_name="typical_sampling",
    metadata_key="typicality",
)

# Sample 5 outliers.
dataset.query().sampling().metadata_weighting(
    n_samples_to_select=5,
    sampling_result_tag_name="outlier_sampling",
    metadata_key="typicality",
    strength=-1
)

TanStack Query v6 is runes-based. To use it in hooks, you must use the .svelte.ts file extension.

Crucial Pattern: For hooks wrapping TanStack Query, accept a getter function (thunk) for reactive parameters. Do not pass Svelte stores or $derived values directly, as TanStack Query v6 requires thunks for reactivity.

Accessing Results: The query result is a reactive proxy. Access properties like query.isSuccess or query.data directly without the $ prefix.

Example: Reactive Hook with Thunk

// useFrames.svelte.ts
export const useFrames = (
  getParams: () => { video_frame_collection_id: string; filter: VideoFrameFilter }
) => {
  const query = createInfiniteQuery(() => {
    const { video_frame_collection_id, filter } = getParams();
    return {
      // ... options
    };
  });
  return { query };
};

// Consumer (+page.svelte)
const { query } = useFrames(() => ({
  video_frame_collection_id: collectionId,
  filter: currentFilter
}));

// Access directly (no $ prefix)
if (query.isSuccess) { ... }

// useFrames.svelte.ts
export const useFrames = (
  getParams: () => { video_frame_collection_id: string; filter: VideoFrameFilter }
) => {
  const query = createInfiniteQuery(() => {
    const { video_frame_collection_id, filter } = getParams();
    return {
      ...getAllFramesInfiniteOptions({
        path: { video_frame_collection_id },
        body: { filter }
      }),
      getNextPageParam: (lastPage) => lastPage.nextCursor || undefined
    };
  });
  return { query };
};

// Consumer (+page.svelte)
const { query } = useFrames(() => ({
  video_frame_collection_id: collectionId,
  filter: currentFilter
}));

Metadata distributions allow you to inspect sample-level attributes. After selecting Metadata in the Distribution panel, you can choose a specific metadata key to visualize.

Numeric Metadata

Numeric fields (e.g., temperature, brightness, confidence, blur score) are displayed as histograms.

• Adjust Detail: Change the number of bins to modify granularity.
• Filtering: Click a bin or drag across multiple bins to filter the dataset to that specific range. Re-selecting the same range clears the filter.

Categorical Metadata

Categorical fields (e.g., city, camera ID, weather, or boolean values) are displayed as bar charts.

• Filtering: Select one or more values to filter the dataset.
• Missing: Represents samples where the selected key has no value.
• Other: Groups less frequent values that are not explicitly shown in the chart (these cannot be selected for filtering).

LightlyStudio Enterprise is designed for scalable ML data management, curation, and annotation. Compared to the Open-Source version, Enterprise provides the following capabilities:

• Multi-user Support: Enterprise supports multiple users with authentication and Single Sign-On (SSO), whereas Open-Source is limited to a single user.
• Dataset Management: Enterprise allows managing multiple datasets with fine-grained access control. Open-Source only supports a single dataset.
• Credential Management: In Enterprise, cloud credentials are centrally managed by an administrator. In Open-Source, they are managed via local environment variables.
• GUI Access: The Enterprise GUI is always running on the server. The Open-Source GUI is started locally using ls.start_gui().
• Python API Usage: The Python API remains the same, but in Enterprise, you must call ls.connect() to establish a connection to the server.

When evaluating a model, high False Positives (fp) or False Negatives (fn) do not always indicate a model failure. Use the following triage logic to determine the cause:

Note: Always re-run evaluation after fixing ground-truth annotations to ensure metrics reflect actual model performance rather than annotation noise.