DOCSEARCHbyAlgolia
Submit Docs

CZ CELLxGENE Annotate

repository·main·Indexed 21 days ago

https://github.com/chanzuckerberg/cellxgene

A web application for the interactive exploration and visualization of large-scale single-cell transcriptomics datasets, capable of handling over 1 million cells. Version 1.3.0 supports Python 3.10+ and allows users to launch the explorer using anndata (.h5ad) files or URLs. The tool includes features for data brushing via histograms, categorical metadata management, and an autosave component for persisting user annotations and genesets to the backend.

Tokens
23.5K
Snippets
81
Records
102
Agent score
74%

What's inside cellxgene

  1. Overview of cellxgene REST API v0.2 routes

    main

    The cellxgene REST API v0.2 uses a versioned prefix for all routes: /api/v0.2/.

    Mandatory Routes:

    • GET /config: Retrieve application configuration.
    • GET /schema: Retrieve data schema for the dataframe and annotations.
    • GET/PUT /annotations/obs & /annotations/var: Fetch or update observation/variable annotation values.
    • GET/PUT /data/obs & /data/var: Fetch or update dataframe values.

    Optional Routes:

    • /cluster/obs & /cluster/var: Fetch cluster assignments or request re-clustering.
    • /layout/obs & /layout/var: Fetch N-dimensional layout coordinates or request re-layout.
    • /diffexp: Fetch differential expression statistics.
    • /saveLocal (or /data/saveSelection): Request the server to persist a filtered data subset to local storage.
    Base URL pattern: /api/v0.2/<route>

  2. Understand the cellxgene REST API 0.2 Data Model

    main

    The cellxgene REST API 0.2 is based on a dataframe model similar to ScanPy's AnnData. The core components are:

    • Dataframe: An expression matrix of shape [nObs, nVar] using IEEE single precision float (float32).
      • Observations (obs): Represent cells. Identified by a matrix coordinate pair (index, base 0).
      • Variables (var): Represent genes or isoforms. Identified by a matrix coordinate pair (index, base 0).
    • Annotations (metadata): Typed metadata attached to either observations or variables.
      • Types: Float32 (JSON number), Int32 (JSON number), String (JSON string), Boolean (JSON true/false), and Categorical (JSON array of enumeration values).
      • Guaranteed Annotation: Every dimension has a unique string annotation named "name" used for display.

    Terminology Mapping:

    cellxgene termbiological/data term
    Observation (obs)Cell
    Variable (var)Gene
    AnnotationMetadata
    DataExpression
    DataframeExpression matrix

  3. Run end-to-end (E2E) tests

    main

    E2E tests (smoke tests) are located in the client directory. To run them, navigate to the client folder and use make smoke-test.

    E2E Test Configuration Flags

    You can control test behavior using the following environment variables:

    • JEST_ENV: Controls the Puppeteer environment (defined in client/jest-puppeteer.config.js).
      • dev: Opens window, minimal slowdown, closes on exit.
      • debug: Opens window, 100ms slowdown, dev tools open, Chrome stays open on exit.
      • prod (default): Headless, no slowdown.
    • HEADFUL: Set to true to launch the Chrome window for visual inspection (default is false).
    • HEADLESS: Set to false to launch the Chrome window for visual inspection (default is true).

    Running E2E tests exactly as CI does

    To replicate the CI environment:

    JEST_ENV=prod make pydist install-dist dev-env smoke-test
    cd client
make smoke-test

# Example with flags
HEADFUL=true npm run e2e

  4. Set up the cellxgene development environment

    main

    To prepare the development environment, run the following from the $PROJECT_ROOT:

    1. Install requirements: make dev-env installs both standard requirements and requirements-dev (needed for building code).
    2. Install cellxgene packages: Use the following commands from $PROJECT_ROOT depending on your needs:
      • install-dev: Installs from the local source tree.
      • install-release-test: Installs from test PyPI.
      • install-release: Installs from PyPI.
      • install-dist: Installs from a local dist folder.
      • uninstall: Uninstalls cellxgene.
    # From $PROJECT_ROOT
make dev-env
install-dev

  5. Install and run the cellxgene client for development

    main

    Client development requires both the server and the client to be running. For hot reloading, they must be launched separately.

    Installation

    1. Install client prerequisites: make dev-env.
    2. Install the cellxgene server using the Server Dev instructions.

    Launching with Hot Reloading

    1. Launch the server: The client requires the REST API. Run cellxgene launch --debug [other_options] <datafile> or make start-server.
    2. Launch the client: Navigate to the client/ directory and run make start-frontend.
    3. The client will be served at localhost:3000.

    Building and Testing

    • Build client only: make build-client.
    • Linting: Uses eslint and prettier.
    • Unit tests: Run make unit-test in the client/ directory.
    • Smoke tests:
      • Standard: make smoke-test.
      • Annotations: make smoke-test-annotations.

    Running Smoke Tests against Hot-Reloaded Client

    1. Start the hot-reloading servers (Server via make start-server, Client via make start-frontend).
    2. If running the standard suite (non-annotations), disable annotations on the backend: CXG_OPTIONS='--debug --disable-annotations' make start-server.
    3. From the client/ directory, run npm run e2e (standard) or npm run e2e-annotations (annotations).
    # 1. Start Server
make start-server

# 2. Start Client (in client/ directory)
cd client
make start-frontend

# 3. Run tests against hot-reloaded client
npm run e2e

  7. Install and run the cellxgene server for development

    main

    To develop on the server side, follow these steps:

    Installation

    • From source tree:
      1. Build client static files: make build-for-server-dev
      2. Install local files: make install-dev
    • From a python distribution:
      1. Create distribution: make pydist
      2. Install distribution: make install-dist

    Launching

    Run cellxgene launch [options] <datafile> or use make start-server.

    Server Reloading

    If installed via make install-dev, the server automatically restarts upon changes to server code. If changes affect the client, you must manually reload the browser.

    Linting and Formatting

    • Auto-format: make fmt (uses black for Python).
    • Lint checks: make lint (uses flake8 for Python).

    Running Server Unit Tests

    1. make dev-env
    2. Run make unit-test inside the server directory, or run make unit-test-server from the project root.
    # Install from source
make build-for-server-dev
make install-dev

# Launch
make start-server

# Lint and Format
make fmt
make lint

  8. How to write a new e2e test

    main

    Follow these steps to implement a new test case:

    1. Validate Scope: Ensure the feature is high-value, stable (not in active development), and depends on the full stack (backend + frontend).
    2. Define Logic: Clearly define the sequence of user actions and the expected outputs.
    3. Instrument Code: Add data-testid (or data-testclass if the element is not unique) to the target input and output elements in the source code.
    4. Implement Test: Create a new test block (optionally nested in a describe block) in client/__tests__/e2e/e2e.test.js.
    5. Verify: Run the tests locally to ensure they pass.

  9. Run end-to-end (e2e) tests

    main

    End-to-end tests are written using the Jest testing framework and automated via Puppeteer. They run against a full-stack version of the web application. You can execute the test suite using either npm or make commands.

    Tests are located in client/__tests__/e2e/e2e.test.js and typically run against the pbmc3k dataset.

    npm run e2e
# OR
make e2e

  10. Run unit tests

    main

    You can run unit tests for the entire project or just the client code.

    For all unit tests:

    1. Start in the project root directory.
    2. Run make dev-env.
    3. Run make unit-test.

    For client code only:

    1. Start in the project root directory.
    2. cd client.
    3. make unit-test.
    # All unit tests
make dev-env
make unit-test

# Client only
cd client
make unit-test

  11. Clean generated files in cellxgene

    main

    Use make commands from the $PROJECT_ROOT to remove generated files or clean the environment:

    • make clean: Performs a full clean, including deleting node_modules (this will make subsequent builds take longer).
    • make clean-lite: Cleans only the built directories.
    • make clean-server: Cleans the server source tree.
    # From $PROJECT_ROOT
make clean
make clean-lite
make clean-server

  12. Run the backend development server with backend_dev

    main

    The ./scripts/backend_dev script is designed for frontend developers to run the REST API required for development without needing deep Python knowledge. It automates creating/activating a virtual environment and installing the current local version of cellxgene.

    Requirements:

    • Python3.10+
    • virtual-env
    • pip

    Usage: Run from the $PROJECT_ROOT.

    Options:

    • Select a dataset: DATASET=<dataset path> ./scripts/backend_dev
    • Pass launch options: Use CXG_OPTIONS to pass flags to the cellxgene launch command. Example: CXG_OPTIONS='--disable-annotations' ./scripts/backend_dev.

    Workflow Tip: You can run ./scripts/backend_dev in one terminal and make start-frontend (from the client directory) in another terminal to develop the full stack.

    # From $PROJECT_ROOT
./scripts/backend_dev

# With a specific dataset and launch options
DATASET=/path/to/data.h5ad CXG_OPTIONS='--disable-annotations' ./scripts/backend_dev