mLLMCelltype

repository·main·Indexed 20 days ago

https://github.com/cafferychen777/mllmcelltype

A multi-LLM consensus framework for automated cell type annotation in single-cell RNA sequencing (scRNA-seq) data. It supports both Python (Scanpy) and R (Seurat) workflows, providing a reference-free approach to reduce single-model bias. The tool features single-model annotation and an iterative multi-LLM consensus process that provides uncertainty metrics such as Consensus Proportion and Shannon Entropy. It supports a wide range of providers including OpenAI, Anthropic, Google, Alibaba, DeepSeek, and others.

Tokens
35.2K
Snippets
105
Records
125
Agent score
69%

What's inside mllmcelltype

  1. What is mLLMCelltype?

    main

    mLLMCelltype is an iterative Multi-LLM consensus framework designed for cell type annotation of single-cell RNA sequencing (scRNA-seq) data. It combines predictions from multiple Large Language Models (including OpenAI GPT-5.5, Anthropic Claude, Google Gemini, etc.) to improve annotation accuracy and provide transparent uncertainty quantification (using consensus ratios and Shannon entropy).

    Key characteristics:

    • No reference dataset required: Unlike traditional methods, it does not need a pre-trained reference.
    • Consensus-based: Reduces errors and hallucinations through multi-round collaborative discussion between models.
    • Integration: Works seamlessly with standard Scanpy (Python) and Seurat (R) workflows.
    • Uncertainty Quantification: Identifies ambiguous cell populations that may require expert review.
  2. Overview of mLLMCelltype

    main
    mLLMCelltype is a multi-LLM consensus framework designed for automated cell type annotation in single-cell RNA sequencing (scRNA-seq) data. It uses a consensus approach where multiple large language models (LLMs) analyze gene expression data to provide predictions, reducing single-model bias and providing uncertainty metrics like Consensus Proportion and Shannon Entropy. The tool is reference-free, meaning it does not require pre-training or reference datasets, and it integrates with common single-cell analysis platforms like Scanpy (Python) and Seurat (R).
  3. Why model quality matters for consensus checking

    main

    The consensus model is responsible for high-level reasoning tasks that directly impact the reliability of cell type annotations. A high-quality model is required to:

    • Evaluate Semantic Similarity: Correctly identify that different names (e.g., "T Lymphocyte" vs "T cell") refer to the same cell type.
    • Understand Biological Context: Grasp biological hierarchies and context.
    • Synthesize Discussions: Aggregate reasoning from multiple models to reach an accurate conclusion.
    • Provide Confidence Metrics: Generate reliable indicators for downstream analysis.

    Using weak models for these tasks can lead to misidentification of controversial clusters, incorrect consensus calculations, and poor resolution of disagreements between models.

  4. Structure of Demo Data Files

    main

    The demo mode relies on two specific files located in notebooks/demo_data/ to simulate the output of the annotation process:

    cached_results.csv

    A simplified CSV containing:

    • Cluster: The cluster identifier (e.g., Cluster_0).
    • Cell Type: The consensus cell type annotation.
    • Consensus Score: Agreement level between models (0-1).
    • Entropy: Shannon entropy measuring annotation uncertainty.

    cached_detailed_results.json

    A complete JSON structure that matches the output of the interactive_consensus_annotation() function. It includes:

    • consensus: Final consensus annotations for each cluster.
    • consensus_proportion: Agreement scores (0-1) for each annotation.
    • entropy: Shannon entropy values.
    • model_annotations: Individual predictions from each model used (e.g., GPT-4 Turbo, Claude Sonnet 4.5, Gemini 1.5 Pro).
    • controversial_clusters: Identifiers for clusters that required multi-round discussion.
  5. How the consensus check model works

    main

    In the consensus framework, the consensus_check_model (R) or consensus_model (Python) is a specialized LLM used to moderate the discussion between other models. It is responsible for:

    1. Evaluating semantic similarity between different cell type annotations (e.g., recognizing "T lymphocyte" and "T cell" are the same).
    2. Calculating consensus metrics like proportion and entropy.
    3. Moderating and synthesizing discussions for controversial clusters.
    4. Making final decisions when models disagree.

    Recommended Models for Consensus Checking:

    • Anthropic: claude-opus-4-7, claude-sonnet-4-6
    • OpenAI: o1, o1-pro, gpt-5.5, gpt-4.1
    • Google: gemini-3.1-pro-preview, gemini-3-flash-preview
    • Other: deepseek-v4-pro, qwen3.6-plus, grok-4.3

    Default Behavior:

    • R: Uses the first model that succeeds during initial annotation if not specified.
    • Python: Selects the consensus checker from providers available in api_keys; pass consensus_model explicitly for reproducible runs.
    # Example: Specifying a consensus check model in R
    consensus_results <- interactive_consensus_annotation(
      input = marker_genes_list,
      tissue_name = "human brain",
      models = c("gpt-5.5", "claude-sonnet-4-6", "gemini-3.1-pro-preview", "qwen3.6-plus"),
      api_keys = api_keys,
      consensus_check_model = "claude-sonnet-4-6",
      controversy_threshold = 0.7,
      entropy_threshold = 1.0
    )
  6. Configure custom API endpoints with base_urls

    main

    Starting from version 1.3.0, you can provide custom endpoints for each API provider using the base_urls parameter in annotate_cell_types() and interactive_consensus_annotation(). This is particularly useful for:

    • Users in China: Accessing international APIs via proxy servers.
    • Enterprise users: Connecting to internal API gateways.
    • Developers: Testing against local or mock API environments.
  7. Smart endpoint selection for Qwen models

    main

    Qwen models support automatic endpoint detection. If you do not provide base_urls, the system will automatically attempt to connect to the international endpoint, then fallback to the domestic endpoint, and finally try the legacy international compatible endpoint. This provides a 'zero-configuration' experience that adapts to your network environment.

    # Recommended usage: Let the system automatically select the best endpoint
    result <- annotate_cell_types(
      input = marker_genes_list,
      tissue_name = "human PBMC",
      model = "qwen3.6-plus",
      api_key = "your-qwen-api-key"
      # No base_urls needed; system auto-selects
    )
  8. How the multi-LLM consensus architecture works

    main

    mLLMCelltype uses a multi-LLM consensus architecture to improve cell type annotation reliability in scRNA-seq data.

    Key components of the workflow include:

    1. Multi-Model Prediction: Multiple LLMs (from providers like OpenAI, Anthropic, Google, etc.) provide initial annotations to reduce individual model bias.
    2. Structured Deliberation: For clusters where models disagree (determined by controversy_threshold and entropy_threshold), the LLMs engage in collaborative, multi-round discussions to refine their answers.
    3. Uncertainty Quantification: The framework explicitly quantifies annotation confidence using consensus proportion and Shannon entropy.
    4. Reference-Free: The process does not require a pre-existing reference dataset, making it applicable to various species and tissues.

    This architecture is designed to integrate seamlessly with Seurat workflows, accepting marker outputs directly.

  9. Set up HTTP proxies in R and Python

    main

    If you are using a local proxy server, you can configure it via environment variables or within your code.

    # R: Set HTTP proxy via environment variables
    Sys.setenv(http_proxy = "http://proxy-server:port")
    Sys.setenv(https_proxy = "http://proxy-server:port")
    
    # R: Set proxy using httr
    library(httr)
    set_config(use_proxy(url = "http://proxy-server:port"))
    # Python: Set HTTP proxy via environment variables
    import os
    os.environ["HTTP_PROXY"] = "http://proxy-server:port"
    os.environ["HTTPS_PROXY"] = "http://proxy-server:port"
    
    # Python: Set proxy using requests
    import requests
    proxies = {
        "http": "http://proxy-server:port",
        "https": "http://proxy-server:port"
    }
    response = requests.get("https://api.example.com", proxies=proxies)
  10. Configure API keys for LLM providers

    main

    mLLMCelltype supports multiple LLM providers (OpenAI, Anthropic, Google, etc.). You must set your API keys as environment variables before running annotations.

    Common environment variables include:

    • ANTHROPIC_API_KEY
    • OPENAI_API_KEY
    • GEMINI_API_KEY
    # Set API keys
    Sys.setenv(ANTHROPIC_API_KEY = "your-anthropic-api-key")
    Sys.setenv(OPENAI_API_KEY = "your-openai-api-key")
    Sys.setenv(GEMINI_API_KEY = "your-gemini-api-key")
  11. Use mLLMCelltype with Seurat objects in R

    main

    In R, mLLMCelltype can be used alongside Seurat to annotate clusters.

    Workflow:

    1. Load your Seurat object.
    2. Identify marker genes using FindAllMarkers.
    3. (Optional) Set a cache_dir to speed up subsequent processing.

    Note: For detailed R tutorials, refer to the official documentation website.

    library(mLLMCelltype)
    library(Seurat)
    
    # Load Seurat object
    pbmc <- readRDS("your_seurat_object.rds")
    
    # Find marker genes
    pbmc_markers <- FindAllMarkers(pbmc,
                                only.pos = TRUE,
                                min.pct = 0.25,
                                logfc.threshold = 0.25)
    
    # Set cache directory
    cache_dir <- "./mllmcelltype_cache"
    dir.create(cache_dir, showWarnings = FALSE, recursive = TRUE)