Epstein Docs Archive

Once data processing and deduplication are complete, use Node.js to generate the searchable static site.

Build the production site:

npm run build

This generates the site in the _site/ directory.

Run a development server:

npm start

This starts a server with live reload capabilities.

Place document images in the downloads/ directory and run the processing script. This script performs OCR, extracts text and entities, and reconstructs multi-page documents. It is resume-friendly and automatically attempts to fix invalid JSON returned by the LLM.

python process_images.py

Options:

• --limit N: Process only N images (useful for testing).
• --workers N: Number of parallel workers (default: 5).
• --no-resume: Process all files, ignoring the existing progress index.

The processing pipeline requires an OpenAI-compatible API. Configure your credentials by copying the example environment file to .env and editing it with your API details.

cp .env.example .env
# Edit .env with your API details

If processing fails or you need to retry specific files, use cleanup_failed.py to manage the processing state.

Check for failures (dry run):

python cleanup_failed.py

Remove failed files from the processed list (to allow retries):

python cleanup_failed.py --doit

Remove failed files and delete corrupt JSON files:

python cleanup_failed.py --doit --delete-invalid-json

This script groups pages into documents and uses AI to generate summaries, key topics, key people, and significance analysis. Results are saved to analyses.json.

python analyze_documents.py

Options:

• --limit N: Analyze only N documents (for testing).
• --force: Re-analyze all documents, ignoring existing analyses.

Use this script to merge inconsistent document type labels (e.g., "deposition" and "DEPOSITION") into canonical forms. This generates a dedupe_types.json file for the website build.

python deduplicate_types.py

To merge entities that have different spellings (e.g., "Epstein" and "Jeffrey Epstein") into a single canonical name, run the deduplication script. This creates a dedupe.json file used by the website build.

python deduplicate.py

Options:

• --batch-size N: Process N entities per batch (default: 50).
• --show-stats: Show deduplication statistics without performing any processing.

To set up the project, you must install dependencies for both the Python OCR processing pipeline and the Node.js static site generator.

Python (OCR processing):

pip install -r requirements.txt

Node.js (website generation):

npm install