DOCSEARCHbyAlgolia
Submit Docs

telegramify-markdown

repository·main·Indexed 18 days ago

https://github.com/sudoskys/telegramify-markdown

A Python library (v1.2.0) to convert Markdown into Telegram-compatible formats. It provides tools to generate (text, entities) tuples to avoid MarkdownV2 escaping issues, produce InputRichMessage payloads for Telegram Bot API 10.1, and handle long messages via automatic splitting. Features include LLM output streaming with DraftStream and EditStream, Mermaid diagram rendering, and LaTeX to Unicode conversion.

Tokens
24K
Snippets
68
Records
92
Agent score
57%

What's inside telegramify-markdown

  1. Core functionality of telegramify-markdown

    main

    The telegramify-markdown library is designed to convert raw Markdown into Telegram-compatible output while preventing the common escaping failures associated with Telegram's MarkdownV2 format.

    Key capabilities include:

    • Markdown to Entities: Converts Markdown into a combination of plain text and a list of Telegram MessageEntity objects.
    • Long Message Handling: An async pipeline that splits long outputs and extracts code blocks or Mermaid diagrams.
    • Fallback Conversion: Converts entity-based output back into MarkdownV2 if the target middleware does not support sending entities.
    • Rich Message Support: Provides optional output for Telegram Bot API 10.1 structured Rich Messages.

  2. Telegram Rich Message limits and block counting

    main

    When using Rich Messages, be aware of Telegram's specific constraints:

    • Text Limit: 32,768 UTF-8 characters.
    • Block Limit: 500 top-level blocks.
    • Nesting Limit: 16 levels of nested formatting/blocks.
    • Media Limit: 50 media attachments.
    • Table Limit: 20 table columns.

    Important: How blocks are counted The 500-block limit applies to top-level block elements as parsed by the Telegram server. Internal children do not increase the top-level count. For example:

    • A single <ul> containing 2,000 <li> items counts as 1 block.
    • A single <table> with 500 rows counts as 1 block.
    • Paragraphs inside a <blockquote> do not add to the top-level block count.

  3. How StreamCore handles concurrency and errors

    main

    The StreamCore class is the underlying mechanism for throttled streaming. It manages a state machine (IDLE → ACTIVE → DONE) and ensures thread-safe-like behavior within a single asyncio event loop.

    Concurrency Guards

    • _sending flag: Prevents overlapping emit or finalize calls. If a timer fires while a previous send is still awaiting, the tick is skipped.
    • finish() logic: Cancels the timer first, awaits any in-flight _sending tasks, then calls finalize.
    • cancel() logic: Cancels the timer and transitions to DONE without calling finalize.

    Error Handling Behavior

    • emit() fails: Logs a warning and skips the update. If it fails 3 times consecutively, the stream enters a "degraded mode" where it stops emitting but continues to accumulate content for the final finalize call.
    • finalize() fails: Propagates the error to the caller (this is critical as final content may be lost).
    • CancelledError: If the wrapping task is cancelled, the stream routes to cancel() instead of finish().

  4. Rich Message chunking and limits

    main

    The telegramify_rich() function handles large content by splitting it into valid chunks.

    • Chunk Constraints: For large inputs (e.g., a 50KB paragraph or code block), the function produces multiple chunks. Each chunk is guaranteed to be within the Telegram limits of 32768 UTF-8 characters and 500 blocks.
    • Tag Integrity: Every chunk produced by telegramify_rich() is guaranteed to be valid Rich HTML, meaning no broken tags or orphaned nesting.
    • Data Serialization: When calling to_dict() on a Rich Message object, optional fields that are None are omitted from the resulting dictionary.

  5. Understand Telegram Rich Message limits and splitting logic

    main

    The library enforces Telegram Bot API 10.1 limits to ensure messages are accepted by the server. Splitting is performed at source block boundaries (e.g., <p>, <ul>, <table>) rather than at arbitrary byte offsets to prevent broken HTML tags or orphaned nesting.

    Enforced Limits

    • Byte Budget: Maximum 32,768 UTF-8 characters per chunk.
    • Block Budget: Maximum 500 top-level blocks per chunk.

    How Block Counting Works

    Telegram counts top-level block elements as parsed by the server. Internal children of a block do not increase the block count. For example:

    • A <ul> containing 1,000 <li> items counts as 1 block.
    • A <table> with 501 rows counts as 1 block.
    • A <blockquote> containing multiple <p> tags counts as 1 block.

    Oversized Atomic Blocks

    If a single atomic element (like a single massive paragraph or a large <pre> code block) exceeds the 32,768-byte limit, the library will split it into multiple chunks while preserving the necessary wrapper tags so that each chunk remains valid HTML.

  6. Understand the streaming draft architecture

    main

    The library uses a two-layer async streaming architecture to handle LLM token-by-token output. This architecture combines a protocol-agnostic StreamCore (responsible for buffering, throttling, and concurrency) with strategy facades like DraftStream or EditStream that handle the actual rendering and Telegram transport.

    Instead of incremental parsing, the library re-parses the entire accumulated buffer on each throttled update. This is because Markdown is context-sensitive (e.g., a character at the end of a string can change the meaning of previous text) and the underlying pyromark parser does not support incremental updates. For typical LLM outputs, the re-parsing cost is negligible compared to network latency.

  7. Understand the core conversion workflows

    main

    The library provides three primary data flow patterns depending on your target Telegram output:

    1. Standard Entity Conversion: Converts raw Markdown into plain text and a list of MessageEntity objects. This is intended for use with the Telegram sendMessage(..., entities=...) method.
    2. Rich Message Conversion: Converts Markdown into an InputRichMessage payload (HTML format). This is intended for use with the Telegram Bot API 10.1 sendRichMessage method.
    3. Pipeline Processing: A high-level async pipeline that processes Markdown and emits a sequence of Text, File, or Photo items. This is designed for callers who want the library to handle long-message splitting and media extraction (like code blocks or Mermaid diagrams) automatically.

  8. Understand the streaming draft support architecture

    main

    The library provides streaming support for Telegram drafts using a two-layer architecture designed to handle the complexities of Markdown rendering during live updates:

    1. StreamCore[T] (Core Layer): A generic, Telegram-agnostic engine that manages the mutable token buffer, throttling (via interval), concurrency guards (preventing overlapping emit calls), and the state machine (handling cancel() and finish()).
    2. DraftStream (Strategy Facade): A high-level layer that applies Telegram-specific logic. It handles render selection (e.g., using richify or convert), thinking_delay (delay before the first emit), sliding window truncation for large messages, and draft_id management.

    Key Mental Model: Fact vs. State

    • State: The accumulated token buffer is an append-only log of tokens. Each emit() output is a temporary projection of this state.
    • Fact: The finalize() output is the final, complete version of the content that is persisted by Telegram.

  9. How Rich Message splitting works

    main

    Splitting Rich HTML is complex because arbitrary byte-offset splitting would break HTML tags and nesting.

    telegramify-markdown uses a source-block-boundary splitting strategy. Instead of splitting the rendered HTML string, it operates on the underlying pyromark source blocks (e.g., <p>, <ul>, <table>, <blockquote>).

    Key Invariants:

    • Atomic Blocks: A block is the smallest unit of splitting. Blocks are never cut in half.
    • Valid Fragments: Each chunk is a valid, self-contained Rich HTML fragment.
    • Limit Awareness: The splitter tracks block count, UTF-8 byte length, and nesting depth per chunk to ensure they stay within Telegram's limits.

    Note on Oversized Blocks: If a single block (like a massive code block) exceeds Telegram's limits on its own, the splitter will emit it as a standalone chunk. The caller must handle potential rejection from the Telegram server in this case.

  10. Technical constraints and requirements for telegramify-markdown

    main

    When integrating telegramify-markdown into your project, be aware of the following technical constraints:

    • Python Version: Requires Python 3.10+.
    • Encoding/Offsets: Telegram MessageEntity offsets and lengths are calculated using UTF-16 code units. This is critical when handling emojis, CJK (Chinese, Japanese, Korean) characters, or mixed text to ensure offsets remain correct.
    • Core Dependencies: The core runtime depends on pyromark.
    • Rich Message Format: Support for Rich Messages follows the Telegram Bot API 10.1 contracts.

  11. Run tests and verification

    main

    To verify the installation and run the test suite, use the following commands:

    # Install test dependencies
pdm install -G tests

# Run standard tests
pdm run test

# Run Rich Message specific tests
pdm run test-rich

# Run live integration tests for Rich Messages (requires Telegram credentials)
TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... pdm run test-live-rich
    pdm install -G tests
pdm run test
pdm run test-rich
TELEGRAM_BOT_TOKEN=... TELEGRAM_CHAT_ID=... pdm run test-live-rich

  12. Use EditStream for streaming edits in group chats

    main

    Use EditStream when you want to send an initial message and then update (edit) it as content arrives. This is ideal for group chats where you want to avoid message spam.

    Key Configuration Options

    • send_message: Async function to send the initial placeholder. It must return the message_id of the sent message.
    • edit_message: Async function to edit the message. Signature: async (message_id, payload) -> None.
    • mode: Either "rich" or "entity".
    • interval: Seconds between edits. Note: Telegram enforces a limit; EditStream enforces a minimum of 1.0s.

    Workflow

    1. On the first emit, send_message is called to create the placeholder.
    2. Subsequent emits call edit_message using the stored message_id.
    3. On finalization, the last edit_message is called with the complete content.
    from telegramify_markdown.stream import EditStream

async with EditStream(
    send_message=my_send_fn,         # async (payload) -> message_id
    edit_message=my_edit_fn,         # async (message_id, payload) -> None
    mode="rich",
    interval=1.0,                    # ≥1.0s enforced (Telegram edit limit)
) as stream:
    await stream.consume(llm_response)