DOCSEARCHbyAlgolia
Submit Docs

pylatexenc

repository·main·Indexed 19 days ago

https://github.com/phfaist/pylatexenc

A simple LaTeX parser providing latex-to-unicode and unicode-to-latex conversion. It includes modules for parsing LaTeX into a node tree (pylatexenc.latexwalker), converting LaTeX to plain Unicode text (pylatexenc.latex2text), and encoding Unicode to LaTeX (pylatexenc.latexencode). The library also provides a mechanism to build a JavaScript version of the pylatexenc.latexnodes library using Transcrypt.

Tokens
13.3K
Snippets
34
Records
70
Agent score
65%

What's inside pylatexenc

  1. New features in pylatexenc 2

    main

    Version 2 of pylatexenc introduces significant improvements to the LaTeX parser, text conversion, and encoding modules:

    LaTeX Parser (pylatexenc.latexwalker)

    • Enhanced Context Management: A more versatile way to provide a "latex context" (macros, environment definitions, and "latex specials") using the pylatexenc.macrospec module.
    • Latex Specials Support: Support for arbitrary character sequences with special LaTeX meanings (e.g., &, #, ``) via the LatexSpecialsNode type.
    • Custom Macro Parsing: Support for arbitrary macro arguments and formats (e.g., \verb+...+ constructs).
    • Math Mode Improvements: Better parsing of math mode, including support for display math modes.
    • Verbatim Representation: LatexNode objects now retain information about their original string position, allowing for reconstruction of their verbatim LaTeX representation.

    LaTeX to Text Conversion (pylatexenc.latex2text)

    • Text Filling: Ability to fill text chunks at a specific column width using the fill_text flag.
    • Whitespace Handling: The strict_latex_spaces flag now defaults to 'macros'.
    • Macro Specification: Renamed classes (e.g., MacroDef is now MacroTextSpec) and added support for "latex specials".
    • Math Mode Conversion: A new math_mode= flag replaces keep_inline_math= for specifying how math mode converts to text.

    Unicode to LaTeX Encoding (pylatexenc.latexencode)

    • New Interface: Introduced UnicodeToLatexEncoder and the unicode_to_latex() function, allowing for custom conversion rules and behavior for unknown characters.
    • Expanded Character Set: Added escapes from the W3C unicode.xml file to the default set.
    • Renaming: utf8tolatex() is deprecated in favor of unicode_to_latex() (though the old function is kept for backward compatibility).

    Command Line Interface

    • The modules latex2text, latexencode, and latexwalker can now be used directly from the command line. Use --help to view available options.

  2. Understand the latexnodes module components

    main

    The pylatexenc.latexnodes module provides tools for building a tree of LaTeX nodes and parsing LaTeX source code. It is organized into several functional areas:

    • Parsing State: Manages the state of the parser (e.g., ParsingState, ParsingStateDelta).
    • Latex Token: Represents individual tokens found in the source (LatexToken).
    • Token Readers: Classes used to iterate through and read tokens (LatexTokenReader, LatexTokenListTokenReader).
    • Arguments: Handles the specification and parsing of command arguments (LatexArgumentSpec, ParsedArguments).
    • Nodes Collector: Responsible for gathering nodes into a tree structure (LatexNodesCollector).
    • Recomposition: Tools to turn a node tree back into LaTeX source code (LatexNodesLatexRecomposer).
    • Exceptions: A hierarchy of error classes for handling parsing and walking failures.

  3. Explore new parsing and tree traversal tools

    main

    Version 3 introduces several new modules and patterns for working with LaTeX node trees:

    • latexnodes module: A new parsing mechanism where everything is delegated to specialized 'parser objects' (e.g., pylatexenc.latexnodes.parsers.LatexParserBase).
    • LatexNodesLatexRecomposer: A tool to turn a node tree back into LaTeX code.
    • LatexNodesVisitor: Implements the visitor pattern for walking a tree. You can use node.accept_node_visitor() to traverse nodes.
    • ParsedArguments: Macro, environment, and specials arguments are now stored in this object. Use pylatexenc.latexnodes.ParsedArgumentsInfo to look up arguments by name for easier traversal.

  4. Whitespace normalization in LatexNodesLatexRecomposer

    main
    When using pylatexenc.latexnodes.LatexNodesLatexRecomposer to recompose a node tree back into LaTeX code, whitespace around paragraph breaks is normalized. Because the specials node records paragraph breaks as exactly \n\n, any existing variations (e.g., 'a\n\n\n\nb') will be normalized to the standard two-newline format (e.g., 'a\n\nb').

  5. Understand Nodes, Node Lists, and Visitors in pylatexenc

    main

    The pylatexenc.latexnodes.nodes module provides the core abstractions for representing a parsed LaTeX document structure.

    • LatexNode: The base class for all individual nodes in the LaTeX tree.
    • LatexNodeList: A container class used to represent a sequence of nodes (e.g., the contents of a group or an environment).
    • LatexNodesVisitor: A base class used to implement the Visitor pattern. You can subclass this to traverse the node tree and perform specific actions (like searching, transforming, or extracting data) without manually iterating through the tree structure.

  6. Manage LaTeX definitions using LatexContextDb

    main

    The LatexContextDb class acts as a 'database' or registry that stores the definitions for macros, environments, and specials. This context is used by the parser to look up how to handle specific LaTeX tokens.

    If you need to extend an existing context with new definitions, you can use ParsingStateDeltaExtendLatexContextDb.

  7. Low-level parsing and macro specification

    main

    The pylatexenc.latexnodes and pylatexenc.macrospec modules provide the low-level parsing layer used by the high-level modules.

    Use these modules if you need to:

    • Teach the parser about custom macros it doesn't recognize.
    • Write your own parser for custom LaTeX constructs.
    • Access the underlying node classes, token reader, or parser object libraries.

  8. Parse LaTeX into a node tree

    main
    If you need to inspect or transform the actual structure of LaTeX code rather than just converting it to text, use the pylatexenc.latexwalker module. It parses LaTeX code into a tree of node objects that represent the document structure.

  10. Define LaTeX macros, environments, and specials with macrospec

    main

    The pylatexenc.macrospec module provides the tools to specify how the parser should interpret LaTeX macros, environments, and special characters. You can define custom behaviors using the following classes:

    • MacroSpec: Defines how a macro (e.g., \command) should be parsed, including its arguments.
    • EnvironmentSpec: Defines how a LaTeX environment (e.g., \begin{env} ... \end{env}) should be parsed.
    • SpecialsSpec: Defines how special characters (e.g., %, $, &) should be handled.

    For standard LaTeX behavior, you can use the helper functions:

    • std_macro()
    • std_environment()
    • std_specials()

  11. Parse LaTeX code and convert to plain text (Unicode)

    main

    To process LaTeX markup, use the following modules:

    1. pylatexenc.latexwalker: Provides routines to parse LaTeX structure into a logical structure of objects. This is intended for parsing chunks of LaTeX as markup rather than acting as a full (La)TeX engine.
    2. pylatexenc.latex2text: Built on top of latexwalker, this module provides functions to convert LaTeX code into plain text using Unicode characters.

    You can also use the latex2text command-line tool to convert LaTeX input (from standard input or files) into plain text via standard output.

  12. Configure math rendering in latex2text

    main

    The latex2text module now uses a new 'fancy' math engine by default (math_mode='fancy'). This engine uses Unicode mathematical alphanumeric characters to mimic LaTeX typesetting (e.g., $x+y$ becomes 𝑥 + 𝑦).

    Potential Issue: These characters are outside the Basic Multilingual Plane and may appear as placeholder boxes in some fonts.

    Solutions:

    1. Restore v2 behavior: Set math_mode='text' in LatexNodes2Text or use the --math-mode=text flag in the CLI.
    2. Keep Unicode but remove styling: Set text_fontstyle=False and math_fontstyle=False to keep Unicode characters but use standard alphanumeric characters.

    Relevant options for pylatexenc.latex2text.LatexNodes2Text include math_mode, math_fontstyle, and math_expression_in.

    # Example of restoring v2 math behavior
from pylatexenc.latex2text import LatexNodes2Text

converter = LatexNodes2Text(math_mode='text')
# ... use converter ...