deepxiv-sdk

repository·main·Indexed 20 days ago

https://github.com/deepxiv/deepxiv_sdk

An agent-first paper search and progressive reading tool for AI agents to find, judge, and read research papers from arXiv, bioRxiv, medRxiv, PMC, and Semantic Scholar. It features a Reader class for programmatic access to paper content, an Agent class for intelligent analysis and literature reviews, and a CLI for searching, trending paper discovery, and web searches. The SDK also supports integration with Claude Desktop via the Model Context Protocol (MCP).

Tokens
16.2K
Snippets
59
Records
70
Agent score
67%

What's inside deepxiv-sdk

  1. How progressive reading works in DeepXiv

    main

    DeepXiv is designed for agents to avoid loading full papers unnecessarily. It follows a "search → judge → read" workflow using specific CLI flags to access content in layers:

    1. Search: Find candidate papers using deepxiv search.
    2. Judge: Use --brief to get a high-level summary (Title, TLDR, keywords, citations, GitHub link).
    3. Analyze Structure: Use --head to see the chapter overview and token distribution.
    4. Deep Read: Use --section NAME to read only specific parts (e.g., Method, Introduction) or use --preview for a ~10k character snippet.
    deepxiv search "agentic memory" --limit 5     # 1. Find candidates
    deepxiv paper 2409.05591 --brief              # 2. Quick judgment
    deepxiv paper 2409.05591 --head               # 3. Check structure
    deepxiv paper 2409.05591 --section Method     # 4. Targeted reading
  2. Use the Agent class for intelligent paper analysis

    main

    The Agent class provides high-level intelligent analysis of research papers. It supports:

    • Simple queries and detailed analysis
    • Follow-up questions (the agent maintains context across queries)
    • Multiple LLM providers (OpenAI, DeepSeek, OpenRouter, etc.)
    • Advanced research tasks like literature reviews, methodology comparisons, and trend analysis

    Key Agent Methods & Tips:

    • agent.reset_papers(): Use this to clear the current context and start fresh on a new topic.
    • print_process=True: Enable this to see the agent's internal reasoning steps.
    • stream=True: Enable this for real-time response streaming.
  3. Customize Agent behavior

    main

    You can fine-tune the Agent's performance by adjusting several parameters:

    • Model Selection: Change the model string (e.g., "gpt-4", "gpt-3.5-turbo", "deepseek-chat").
    • LLM Parameters: Adjust temperature, max_tokens, and max_llm_calls to control creativity and cost/depth.
    • Providers: Switch between different LLM providers like OpenAI, DeepSeek, or OpenRouter.
  4. Install deepxiv-sdk

    main

    You can install the core SDK using pip. For a full-stack experience including an MCP server and a built-in research agent, install the all extra.

    On first use, the CLI automatically registers a free anonymous token (1,000 requests/day) and saves it to ~/.env, so no manual setup is required before your first query.

    # Standard installation
    pip install deepxiv-sdk
    
    # Full stack installation (includes MCP server + research agent)
    pip install "deepxiv-sdk[all]"
  5. Perform batch processing and paginated searches

    main

    For large-scale tasks, use pagination and batch loops.

    Pagination: Use the offset parameter in reader.search() to iterate through results in chunks.

    Example: Paginated Search

    all_results = []
    for offset in range(0, 500, 100):
        results = reader.search("agent memory", size=100, offset=offset)
        all_results.extend(results['results'])
    # 获取前 500 个结果
    all_results = []
    for offset in range(0, 500, 100):
        results = reader.search(
            "agent memory",
            size=100,
            offset=offset
        )
        all_results.extend(results['results'])
    
    print(f"总共获取论文数: {len(all_results)}")
  6. Manage DeepXiv Tokens

    main

    DeepXiv resolves tokens in the following order: --token option $\rightarrow$ DEEPXIV_TOKEN environment variable $\rightarrow$ ~/.env file.

    Token Types and Limits

    TypeDaily LimitHow to get
    Anonymous (Auto-registered)1,000 requestsAutomatically happens on first CLI use
    Registered Token10,000 requestsRegister here
    Custom / HigherContact supportEmail tommy[at]chien.io

    Commands

    # Automatically register (recommended for first use)
    deepxiv search "agent"
    
    # Save a token to ~/.env
    deepxiv config --token YOUR_TOKEN
    
    # Use an environment variable
    export DEEPXIV_TOKEN="your_token"
    
    # Pass token per command
    deepxiv paper 2409.05591 --token YOUR_TOKEN
  7. Integrate DeepXiv as an MCP Server for Claude

    main

    You can add DeepXiv to Claude Desktop as an MCP (Model Context Protocol) server. This allows Claude to use tools like search_papers, get_paper_brief, and get_full_paper directly.

    Add the following configuration to your claude_desktop_config.json:

    Paths:

    • macOS: ~/Library/Application Support/Claude/claude_desktop_config.json
    • Windows: %APPDATA%\Claude\claude_desktop_config.json
    • Linux: ~/.config/Claude/claude_desktop_config.json
    {
      "mcpServers": {
        "deepxiv": {
          "command": "deepxiv",
          "args": ["serve"],
          "env": {
            "DEEPXIV_TOKEN": "your_token_here"
          }
        }
      }
    }
  8. Use the DeepXiv Research Agent

    main

    The CLI includes a built-in ReAct agent capable of multi-turn search, reading, and reasoning. It supports any OpenAI-compatible API.

    CLI Usage:

    1. Install with extras: pip install "deepxiv-sdk[all]"
    2. Configure your LLM: deepxiv agent config
    3. Run a query: deepxiv agent query "Your question"

    Python Usage: Use the Agent class for programmatic research workflows.

    Important Note on Reasoning Models: If using models like DeepSeek-R1 or MiMo, you must disable thinking for multi-round tool use to avoid errors. Use enable_thinking=False in the constructor or the --disable-thinking flag in the CLI.

    from deepxiv_sdk import Agent
    
    # Configure for a specific provider
    agent = Agent(
        api_key="your_key", 
        base_url="https://api.deepseek.com/v1", 
        model="deepseek-chat",
        enable_thinking=False
    )
    
    print(agent.query("Compare key ideas in transformers and attention mechanisms"))
  9. Configure API tokens

    main

    The SDK requires API tokens for DeepXiv and your chosen LLM provider.

    DeepXiv Token: The deepxiv CLI auto-registers DEEPXIV_TOKEN on its first use and saves it to ~/.env. You can also set it manually as an environment variable.

    LLM Provider Tokens: Set the appropriate environment variable for your provider (e.g., OPENAI_API_KEY or DEEPSEEK_API_KEY).

    export DEEPXIV_TOKEN="your_deepxiv_token"
    export OPENAI_API_KEY="your_openai_key"
    # Or for DeepSeek:
    export DEEPSEEK_API_KEY="your_deepseek_key"
  10. Use the Python SDK for paper search and reading

    main

    The Reader class is the primary interface for interacting with DeepXiv. You can use it to search for papers across different sources (defaulting to arXiv) and perform progressive reading (from brief summaries to full markdown content).

    Search Workflow

    Use reader.search() to find papers. You can filter by source, categories, authors, organizations, venues, and date ranges.

    Progressive Reading Workflow

    Once you have an arxiv_id (or equivalent ID from other sources), you can extract information at increasing levels of detail:

    1. brief(arxiv_id): Get title, TLDR, keywords, citation count, and GitHub link.
    2. head(arxiv_id): Get metadata and a chapter overview.
    3. section(arxiv_id, name): Get a specific section (e.g., "Introduction").
    4. raw(arxiv_id): Get the full paper in markdown format.
    from deepxiv_sdk import Reader
    
    reader = Reader()
    
    # Search for papers
    results = reader.search("agent memory", size=5)
    for paper in results["result"]:
        print(paper["arxiv_id"], paper["score"], paper["title"])
    
    # Progressive reading
    brief = reader.brief("2409.05591")
    head = reader.head("2409.05591")
    intro = reader.section("2409.05591", "Introduction")
  11. Create a Trending Paper Digest

    main

    Use the deepxiv-trending-digest workflow to generate a concise markdown report of recent hot academic papers. This process follows a progressive reading strategy: searching for trending papers, briefing candidates, inspecting structure, and performing targeted section reads to avoid unnecessary full-text processing.

    Workflow Steps

    1. Pull trending papers: Identify recent hot papers using the trending command.
    2. Brief candidates: Use the --brief flag to get a high-level summary (title, TLDR, keywords, etc.) for each paper to screen them.
    3. Rank and Select: Choose 1-3 promising papers for deeper inspection based on novelty, relevance, or momentum.
    4. Inspect structure: Use the --head flag on selected papers to view their section headers and decide which parts are worth reading.
    5. Targeted section reads: Use the --section flag to read only high-value parts (e.g., Introduction, Method, Results) rather than the entire paper.

    When writing the digest, include an Executive Summary, a list of Papers Reviewed (distinguishing between those only briefed and those deeply inspected), a Recommended Deep Dives section, and Cross-Cutting Trends.

    # 1. Find trending papers
    deepxiv trending --days 7 --limit 10 --json
    
    # 2. Brief a candidate
    deepxiv paper <arxiv_id> --brief
    
    # 3. Inspect structure of a promising paper
    deepxiv paper <arxiv_id> --head
    
    # 4. Read a specific high-value section
    deepxiv paper <arxiv_id> --section Introduction