DOCSEARCHbyAlgolia
Submit Docs

Substack2Markdown

repository·main·Indexed 19 days ago

https://github.com/timf34/substack2markdown

A Python-based tool to download free and premium Substack posts and convert them into Markdown and HTML formats. It features a CLI for scraping, support for MDX-compatible frontmatter, local image downloading, and a built-in HTML browser for organizing and viewing scraped content.

Tokens
1K
Snippets
2
Records
7
Agent score
18%

What's inside substack2markdown

  3. Install Substack2Markdown

    main

    To install Substack2Markdown, clone the repository and install the required Python dependencies using pip. It is recommended to use a virtual environment.

    Prerequisites:

    • Python installed
    • Microsoft Edge installed (required for the Selenium webdriver used in premium scraping)

    Steps:

    1. Clone the repository.
    2. (Optional) Create and activate a virtual environment.
    3. Install dependencies via requirements.txt.
    git clone https://github.com/yourusername/substack_scraper.git
cd substack_scraper

# Optional: Create and activate virtual environment
python -m venv venv
# Windows:
.\venv\Scripts\activate
# Linux:
$ source venv/bin/activate

pip install -r requirements.txt

  4. Use the Substack Reader online version

    main

    For a version that requires no installation, you can use the web-based Substack Reader.

    Capabilities:

    • View content or export Markdown files directly in the browser.
    • Works for free Substacks only.
    • Does not support premium content export.

    How to use:

    1. Visit the website.
    2. Enter the Substack URL.
    3. Click "Go" to view or "Export" to download.

  5. View scraped Markdown files in a browser

    main

    After running the script, files are saved in /substack_md_files (Markdown) and /substack_html_pages (HTML). To view the content:

    1. HTML Interface: The script generates an HTML file in the /substack_html_pages directory. You can browse and sort essays by date or likes using this interface.
    2. Markdown Viewer Extension: Install a browser extension like Markdown Viewer to view .md files directly.
    3. Online Tool: Use the Substack Reader web tool for free content.

  6. Configure premium content scraping

    main

    To scrape premium (paid) Substack content, you must provide your Substack credentials. Update the config.py file in the root directory of the project with your account details.

    Note: You must be an active subscriber to the Substack you are attempting to scrape to access paid content.

    EMAIL = "your-email@domain.com"
PASSWORD = "your-password"

  7. Use the Substack2Markdown CLI

    main

    The substack_scraper.py script can be run with various arguments to control the scraping process. You can either hardcode the URL and post count in the script or pass them via the command line.

    Common Commands

    Scrape a free Substack:

    python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts

    Scrape a premium Substack: Requires config.py to be set up and Microsoft Edge installed.

    python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts --premium

    Scrape a single post:

    python substack_scraper.py --url https://example.substack.com/p/my-post

    Download images locally: Downloads Substack-hosted images and rewrites the Markdown links to point to local files.

    python substack_scraper.py --url https://example.substack.com --images

    Limit the number of posts:

    python substack_scraper.py --url https://example.substack.com --directory /path/to/save/posts --number 5

    Emit MDX-compatible frontmatter: Instead of standard headers, this uses YAML frontmatter (title, subtitle, date, author, image) suitable for MDX sites.

    python substack_scraper.py --url https://example.substack.com --frontmatter mdx