RefChecker Documentation

repository·main·Indexed 19 days ago

https://github.com/markrussinovich/refchecker

A tool for validating reference accuracy in academic papers. RefChecker detects fabricated references and metadata mismatches by cross-referencing citations against databases like Semantic Scholar, OpenAlex, CrossRef, DBLP, and ACL Anthology, using LLMs for deep web verification. It is available via Web UI, Desktop App (Tauri), CLI, and HTTP API. Key features include hallucination checks, AI-generated-text detection, retraction screening, and a native PDF viewer with two-way linking.

Tokens
61.1K
Snippets
171
Records
281
Agent score
65%

What's inside RefChecker

  1. Overview of the Button & Action-Area Redesign Spec (v2)

    main

    The Button & Action-Area Redesign Spec (v2) defines the visual and behavioral requirements for controls in the refchecker web-ui article-level action area. The goal is to achieve visual consistency, click-state stability, and proper grouping/rhythm across all interactive elements.

    Key Requirements

    • R1 — Visual consistency: All pills (retraction status, headers, assistant triggers, AI pills) must share a single token system for height, padding, radius, fonts, and borders.
    • R2 — Click-state stability: Controls must not mutate their geometry (width, height, border-radius, or border) when clicked. Only inner content (e.g., swapping an icon for a spinner) is permitted to change.
    • R3 — Grouping & rhythm: Controls must align on a shared baseline, share vertical rhythm, and follow defined max-width and caption placement rules.

    Scope of Affected Components

    This spec applies to the controls located between the Stats/health row and the References/Corrections tabs in web-ui/src/components/MainPanel/MainPanel.jsx:

    1. Retraction status pill (RetractionCheck.jsx)
    2. Gap-finder collapsible header (GapFinder.jsx)
    3. Citation-numbering split-button (CitationIntegrity.jsx)
    4. Article assistant (ArticleAssistant.jsx)
    5. AI-likelihood expandable row (AIDetectionPanel.jsx)
  2. RefChecker Access Methods and Core Engine

    main

    RefChecker provides four distinct access methods that all utilize the same underlying verification engine (backend.refchecker_wrapper.ProgressRefChecker).

    Access Methods

    • Web UI: A React SPA served by a FastAPI backend (refchecker-webui).
    • Desktop (Tauri): A native macOS, Windows, or Linux application bundling the Web UI and a PyInstaller sidecar backend.
    • CLI:
      • academic-refchecker: Used for standalone batch processing and report generation.
      • refchecker-webui check: Used for single-paper checks with web-parity.
    • API: FastAPI HTTP endpoints used by the Web UI.

    Core Principles

    • Honesty: RefChecker does not fabricate data. If a source cannot be resolved, the system abstains (emits no badge/verdict) rather than providing an incorrect one.
    • AI Detection: AI-generated-text detection is advisory and opt-in; it is not intended as definitive proof of misconduct.
    • Privacy/Mode: All surfaces run in single-user/local mode by default. Multi-user features (Accounts, Teams, shared presence) must be explicitly enabled via multi-user mode.
  3. RefChecker Desktop features

    main

    The desktop application extends the web UI with several native-optimized features:

    • Native PDF Viewers: Includes status color coding, in-document search (⌘F), citation links (reference ↔ document), and trackpad pinch-zoom.
    • Visualizations: Radial graphs for seen libraries and 3D force-directed graphs for reference history.
    • AI Capabilities: Per-reference chat and summarization grounded in article text, plus opt-in AI-generated text detection.
    • Export Options: Self-contained HTML, publish links, Markdown, PDF, DOCX, RIS, and animated video walkthroughs.
    • Collaboration: Opt-in support for Accounts, Teams, and presence (can be enabled via hot-reload).
    • Deployment: Auto-updating, signed installers for all major desktop platforms.

    Note: All features are single-user/local by default.

  4. Explore RefChecker access methods

    main

    RefChecker provides a single verification engine accessible through four primary interfaces. Depending on your workflow, you can use:

    • Web UI: For browser-based interaction.
    • Desktop App (Tauri): For a native application experience.
    • CLI: For command-line automation and single-paper checks using refchecker-webui check.
    • HTTP API: For programmatic integration into other systems.

    For a detailed breakdown of which features (such as hallucination checks, gap-finding, or enrichment) are available in each interface, refer to the FEATURES.md guide.

  5. Align elements within a block

    main

    Each functional block should follow a two-row vertical structure to maintain consistent alignment between controls and their descriptive text.

    Block Structure:

    • Row 1 (Control): Contains the pill, split-button, or header. It should have an intrinsic width and a height of 28px.
    • Row 2 (Information): Contains either a caption bubble or a result panel.

    Styling Rules:

    • Caption Bubbles: Positioned directly under the pill with a gap: var(--control-caption-gap) (4px). They must be left-aligned to the pill, use text-xs, and have color: var(--color-text-muted) with no border or background.
    • Result Panels: (e.g., gap lists, numbering details) Should remain full column width using the existing rounded-lg p-3 card treatment.
    /* Conceptual Block Layout */
    <div style="display: flex; flex-direction: column; gap: var(--control-caption-gap); align-items: flex-start;">
      <div class="pill">Control</div>
      <div class="caption-bubble" style="font-size: var(--text-xs); color: var(--color-text-muted);">Caption text...</div>
    </div>
  6. How the Hallucination Check flow works

    main

    The hallucination check is a deep verification step performed after standard database lookups. It is designed to catch references that look plausible but are actually fabricated.

    The Process

    1. Deterministic Filtering: RefChecker first applies filters to avoid unnecessary LLM costs for minor issues (e.g., year-only mismatches).
    2. Triggering: Suspicious cases (unverified references, low author overlap, identifier conflicts, or URLs resolving to different works) are sent to the hallucination LLM.
    3. LLM Web Search: The LLM performs a web search to find the actual source of the citation.
    4. Structured Verdict: The LLM returns a verdict, explanation, and metadata (title, authors, year, link).
    5. Re-verification: If the LLM finds a source (UNLIKELY verdict), RefChecker re-runs standard comparisons against the LLM-found metadata. This can clear false positives caused by stale database entries.
    6. Final Result: If no source is found or the source conflicts with the citation, it is marked as a likely hallucination.
  7. How the Auto-Update Flow Works

    main

    The auto-update mechanism follows these steps during a release:

    1. Tagging: Pushing a tag like desktop-v0.1.0 triggers the workflow.
    2. Version Sync: The workflow extracts the version from the tag and updates tauri-app/package.json. Tauri uses this version for the bundled app and the updater.
    3. Artifact Creation: The workflow builds platform-specific artifacts (e.g., .app.tar.gz, .msi.zip) and generates a .sig signature for each using the TAURI_SIGNING_PRIVATE_KEY.
    4. Manifest Assembly: A latest.json manifest is created, containing the URLs and signatures for all platform artifacts.
    5. Release: The artifacts and latest.json are uploaded to a GitHub Release.
    6. Polling: Installed apps poll the GitHub release URL for latest.json. If the version in the manifest is higher than the running version, the app downloads the bundle, verifies the signature against the embedded public key, applies the update, and restarts.
  8. Understand RefChecker test categories

    main

    The test suite is divided into three main categories:

    1. Unit Tests (tests/unit/): Fast regression tests for parsing, matching, policy, API guards, and helpers (e.g., text and citation normalization).
    2. Integration Tests (tests/integration/): Broader workflow tests using cached fixtures or live dependencies. Note that tests/integration/test_grobid_integration.py requires Docker and local PDF fixtures.
    3. End-to-End Tests (tests/e2e/): Reserved for browser or workflow tests; currently minimal.
  9. Prevent label reflow using the invisible sizer-grid technique

    main

    To prevent UI 'jumping' or reflow when a control's label changes (e.g., from 'Checking...' to a result string), use the invisible longest-label sizer technique. This is the mandatory method for status pills like RetractionCheck.jsx and CitationIntegrity.jsx.

    Implementation Steps:

    1. Define an array LABELS containing every possible string the control can display.
    2. Use an inline-grid container where all labels (both the sizer and the live label) occupy the same grid cell (1 / 1).
    3. Render the sizer labels with visibility: hidden and white-space: nowrap. The widest label will set the container's width.
    4. Render the live label on top of the sizer.

    This ensures the button width is determined by the longest possible string and remains constant regardless of the current state (rest, loading, or result).

    // LABELS = every string this control can show, longest decides the width.
    <span style={{ position: 'relative', display: 'inline-grid' }}>
      {/* sizer: every candidate stacked in the SAME grid cell; the widest one
          sets the box width; all are visually hidden but occupy space */}
      {LABELS.map((t) => (
        <span key={t} aria-hidden
          style={{ gridArea: '1 / 1', visibility: 'hidden', whiteSpace: 'nowrap' }}>
          {t}
        </span>
      ))}
      {/* live label overlaid in the same cell */}
      <span style={{ gridArea: '1 / 1', whiteSpace: 'nowrap', textAlign: 'left' }}>
        {btnLabel}
      </span>
    </span>
  10. Use the Article Tools in the Desktop App

    main

    The desktop application provides four on-demand article tools organized in a 2×2 button grid. Clicking a tool opens its details full-width directly below the grid:

    • Retractions: Check for retracted papers.
    • Gap-finder: Identify missing or incomplete citations.
    • Citation-numbering: Check for gaps, out-of-order, duplicates, undefined, or uncited references (supports various numbering schemes).
    • Chat & Summarize: Use LLMs to chat about a reference or summarize text (grounded in fetched full text when open access allows).
  11. Configure AI Detection Run Modes

    main

    When running checks, you can specify the detection run-mode to focus on specific tasks:

    • References only: Focus on verifying the reference list.
    • AI detection only: Focus on identifying AI-generated text.
    • Both: Run both reference verification and AI detection simultaneously.