Texy PHP Text-to-HTML Engine

repository·master·Indexed 41 days ago

https://github.com/dg/texy

A PHP-based text-to-HTML engine for lightweight markup languages, focusing on structurally valid HTML and high-quality typography. It utilizes an Abstract Syntax Tree (AST) and a four-phase processing lifecycle (Normalize, Parse, Transform, Render). Texy supports custom syntax handlers, renderers, and a safe mode for processing untrusted user input. A companion package, texy/markdown, provides full CommonMark and GitHub Flavored Markdown (GFM) support.

Tokens
27.4K
Snippets
77
Records
113
Agent score
100%

What's inside Texy

  1. Overview of Texy documentation

    master

    Texy is a markup-to-HTML converter. The documentation is organized into two main sections: Reference, which covers using and extending the library, and Internals, which explains the underlying conversion pipeline.

    To use Texy, you should consult the following guides:

    • Syntax: For the complete reference of the Texy markup language.
    • Configuration: For all configuration options, including Texy properties, generator settings, and security (safe mode).
    • Extending: For instructions on adding custom handlers (renderers, transformation passes, or directives) and custom syntax (line or block syntax).
  2. Configure Texy via public properties

    master

    Texy follows a semantic/presentation split for configuration. You configure the engine through three main areas:

    1. The Texy class: Controls which syntaxes are active (e.g., images, emoticons).
    2. Modules: Semantic configuration (e.g., heading levels, typography locale).
    3. The HTML generator ($texy->htmlOutput): Presentation and output formatting (e.g., image roots, line wrapping, HTML tag whitelists).

    Note: Accessing pre-4.0 property locations (like $texy->imageModule->root) is deprecated and will trigger E_USER_DEPRECATED.

    $texy = new Texy\Texy;
    $texy->allowed['emoticon'] = true;             // Texy: syntax + security
    $texy->typographyModule->locale = 'en';        // module: semantic configuration
    $texy->htmlOutput->imageRoot = '/images/';  // generator: presentation
    $texy->htmlOutput->lineWrap = 120;       // output formatting
  3. Understand Texy terminology and core concepts

    master

    To extend or customize Texy, you must understand its core abstractions:

    • Syntax: A named syntactic construct (e.g., phrase/strong or image). Names are constants on Texy\Syntax and serve as keys in Texy::$allowed. Hierarchies are used for related groups (e.g., phrase/*, block/*).
    • Pattern: The regular expression used to recognize a syntax. Patterns are registered via Texy::addLineSyntax() or addBlockSyntax() along with a syntax name and a parser.
    • Syntax handler: A closure called by the parser when a pattern matches. It has the signature function(ParseContext $context, array $matches, array $offsets, string $name): ?Node. It must return an AST node or null to refuse the match.
    • Transform pass: A callable registered via Texy::addPass() that receives the DocumentNode after parsing. This is used to mutate the AST, typically using NodeHelpers or NodeTraverser.
    • Node renderer: A callable registered on an output generator via NodeRenderer::addRenderer() to handle specific node classes during the rendering phase.
  4. How typography and hyphenation work in Texy

    master

    Typography (locale-aware quotes, dashes, non-breaking spaces) and long-word hyphenation are implemented as AST transformations via Texy\TextRunPass, not as string post-processing. This ensures that every output format (HTML, Text, etc.) receives the same typographic treatment.

    Mechanism: For every block-level inline container, Texy builds a "text image" by joining TextNode contents with special marker characters (\x17, \x16, \x15) that represent markup boundaries and protected text (like code or URLs). Regex transformers run over this image, treating the markers as a transparent alphabet. This allows the transformation to be split back into the original text nodes unambiguously.

    Configuration: Control these features using the $allowed array:

    • $allowed['typography']
    • $allowed['longwords']
  5. Configure security policies with HtmlPolicy and UrlPolicy

    master

    Security is handled via policies exposed on the Texy object. These policies drive the sanitization pass, which modifies the AST to protect every output format (not just HTML).

    • Texy\UrlPolicy: Used for filtering URL schemes.
    • Texy\HtmlPolicy: Accessed via $texy->htmlPolicy. It manages whitelists for:
      • Tags ($allowedTags)
      • Classes ($allowedClasses)
      • Styles ($allowedStyles)
      • Passthrough tag validation (e.g., ensuring <a> has an href and a safe scheme).

    When a tag is rejected by the policy, the HtmlSanitizePass reconstructs the rejected tag as visible TextNodes, ensuring the content is safe but still readable.

    Note: The HtmlPolicy is used both during the transform phase (as the authoritative decision) and by the renderer as defense-in-depth.

  6. How InlineParser resolves syntax collisions

    master

    When multiple inline patterns match the same text, InlineParser resolves collisions using a specific priority hierarchy:

    1. Leftmost match wins: The pattern that starts at the earliest offset is processed first.
    2. Longest match wins: If multiple patterns start at the same offset, the one that matches the most characters is chosen.
    3. Registration order: If matches have the same offset and length, the pattern registered first takes precedence.

    Handler Return Values

    Syntax handlers can return different types of results:

    • InlineNode or BlockNode: A single node representing the match.
    • ContentNode: A sequence of nodes. Its children are spliced into the result in place of the single match. Returning the result of $context->parseInline() directly performs a splice rather than a nest.
    • null: The handler refuses the match. This allows other patterns (including overlapping ones) a chance to claim the text. This is a powerful way to implement context-sensitive priority without a global priority table.
  7. Best practices for custom Texy syntax

    master

    When extending Texy, follow these guidelines to ensure compatibility and stability:

    • Refusal: Return null if your pattern matches but shouldn't be applied. This allows other syntaxes to attempt a match.
    • Node Types: Prefer existing node types (e.g., LinkNode) to get automatic rendering in all output formats. If you must create a new Node subclass, implement getChildren() and provide a renderer handler.
    • Namespacing: Use prefixed names (e.g., myapp/alert) to avoid collisions and allow users to disable your syntax via $texy->allowed.
    • Modifiers: Support Texy modifiers by adding Texy\Patterns::Modifier . '?' to your regex and using Texy\Modifier::parse($matches[n]).
    • Metadata: Fill in the Texy\Range object where practical to support editor tooling and source position tracking.
  8. Extend Texy using Transform passes

    master

    Transform passes allow you to mutate the Abstract Syntax Tree (AST) after parsing but before rendering. You register a pass using Texy::addPass().

    Key behaviors:

    • The pass receives a DocumentNode representing the entire parsed document.
    • You can use NodeHelpers for simple mutations or NodeTraverser for structural changes.
    • All registered passes run in the order they were registered.
    • Important: Passes registered by modules always run before consumer-added passes. If you need to ensure a specific order of operations, manually call parse(), perform your AST manipulations, and then call renderHtml().
  9. How modifiers are applied to elements

    master

    Modifiers are semantic parts of an AST node and are applied during the rendering phase via Output\Html\ElementDecorator. The application process respects the htmlPolicy configuration to ensure safe output:

    1. Attributes: Only attributes allowed by the $htmlPolicy->allowedTags configuration are copied.
    2. Title: Applied and typographically corrected.
    3. Classes and ID: Filtered by $texy->htmlPolicy->allowedClasses. If a whitelist is used, only listed classes are applied. An ID is only applied if the whitelist contains '#' . $id.
    4. Styles: Filtered by $texy->htmlPolicy->allowedStyles.
    5. Alignment: Applied either as a CSS class (defined in $htmlOutput->alignClasses) or as inline text-align / vertical-align styles.
  10. How modules interact with each other

    master

    Modules in Texy do not communicate directly with each other; instead, they interact through the Abstract Syntax Tree (AST). This ensures modules remain replaceable and relations are primarily one-directional.

    Key interaction patterns:

    • Nodes as the interface: Modules express relationships by nesting nodes. For example, a clickable image is a LinkNode wrapping an ImageNode, and a figure holds its image node. There are no shared mutable value objects.
    • Reference resolution via the transform phase: Definitions collected during parsing (like ImageDefinitionNode or LinkDefinitionNode) are stored in the tree. During the afterParse phase, modules use NodeHelpers::find() to walk the document and fill in the referencing nodes.
    • Persistence: User-supplied definitions (via addDefinition()) persist across multiple process() calls.
    • Directionality: Relations are one-directional. For example, PhraseModule knows LinkNodes exist, but link modules do not need to know about phrases.
  11. Use Modifiers for Styling

    master

    Modifiers can be applied to almost every Texy construct to add classes, IDs, or inline styles.

    The modifier syntax follows this pattern: .(title)[class #id]{style: value; attr: value}<>^

    Example usage:

    • .(class) for a class
    • [#id] for an ID
    • {style: color: red;} for inline styles

    Note: The actual allowed classes and styles are restricted by the $htmlPolicy->allowedClasses and $htmlPolicy->allowedStyles configurations.

  12. Behavioral changes in Texy 4

    master

    When upgrading to Texy 4, be aware of the following changes in how text is processed:

    • Tabs: Tabs are no longer expanded to spaces; they count as a single whitespace character. The $tabWidth property is removed.
    • Soft Hyphens: These are now preserved in text and only stripped from URL targets. The $removeSoftHyphens option is removed.
    • Input Normalization: Texy::normalize() is now a static, idempotent method that unifies line endings, removes the leading BOM, control characters, and trailing whitespace. It runs automatically as the first step of every entry point. The preprocess() step has been removed.
    • UTF-8 Validation: Invalid UTF-8 input now throws a Texy\InvalidArgumentException immediately.
    • Metadata and Headings: Headings, TOC, and metadata are stored within the document tree rather than in module state. To access them, use HeadingNode::collectFrom($document) on the AST returned by parse().