Literate.jl

repository·master·Indexed 18 days ago

https://github.com/fredrikekre/literate.jl

A literate programming tool for Julia that enables developers to write code and prose in a single .jl file and export it to Markdown, Jupyter notebooks, and plain Julia scripts. It supports custom pre- and post-processing functions, line filtering via output tokens, and integration with Documenter.jl. The tool provides multiple Markdown flavors, including Documenter, CommonMark, Franklin, and Quarto (.qmd).

Tokens
5.9K
Snippets
23
Records
31
Agent score
69%

What's inside Literate.jl

  1. What is Literate and how does it work?

    master

    Literate is a package designed for Literate Programming. It allows you to write a single commented Julia script as your source of truth and automatically generates multiple output formats from it. This ensures that your documentation, interactive notebooks, and executable scripts stay in sync with minimal maintenance.

    By using a single source file, you can provide users with different ways to consume your package's examples: markdown for reading, notebooks for interactive exploration, and plain scripts for direct execution.

  2. Use custom pre- and post-processing functions

    master

    Literate allows you to hook into the generation process to transform content using preprocess and postprocess keyword arguments. This is useful when standard line filtering is insufficient for your documentation needs.

    All generators (Literate.markdown, Literate.notebook, and Literate.script) support these arguments:

    • preprocess: Receives the raw input String read from the source file (after default line-ending transformations). It must return a transformed String.
    • postprocess:
      • For markdown and script output: Receives the final content String just before it is written to the file. It must return a transformed String.
      • For notebook output: Receives the dictionary representing the notebook.

    The default transformation for both is the identity function.

    # Example usage pattern
    Literate.markdown("input.jl", "outputdir"; preprocess = my_preprocess_func, postprocess = my_postprocess_func)
  3. Configure Literate output behavior

    master

    The functions Literate.markdown, Literate.notebook, and Literate.script can be configured using keyword arguments or a config::Dict object.

    Precedence Rules:

    1. Individual keyword arguments (highest priority).
    2. The config dictionary.
    3. Literate.DEFAULT_CONFIGURATION (lowest priority).

    Example of precedence: Literate.markdown(..., config = Dict("name" => "hello"), name = "world") will result in the name being "world".

  4. Add cell metadata to Notebooks using %%

    master

    You can inject Jupyter metadata (for tools like nbgrader or RISE) into cells using the %% token. The syntax is:

    %% optional ignored text [type] {optional metadata JSON}

    To ensure metadata is only included in the notebook output and not in other formats, prefix the line with #nb:

    #nb %% A slide [markdown] {"slideshow": {"slide_type": "slide"}}
    # # Some title
    
    #nb %% A slide [code] {"slideshow": {"slide_type": "fragment"}}
    x = 1//3
    #nb %% A slide [markdown] {"slideshow": {"slide_type": "slide"}}
    # # Some title
  5. Understand the Literate processing pipeline

    master

    Literate follows a consistent five-step pipeline to transform input files into various output formats:

    1. Pre-processing: The input file is read as a string. User-defined pre-processing functions are applied, followed by built-in replacements (e.g., converting CRLF to LF), line filtering (handling #md , #nb , or #jl tokens), and macro expansion.
    2. Parsing: Lines are categorized as either markdown or code. Adjacent lines of the same type are grouped into "chunks". Leading/trailing empty lines in chunks are removed, and leading # tokens are stripped from markdown chunks.
    3. Document generation: Chunks are transformed based on the target format (Markdown, Notebook, or Script).
    4. Post-processing: User-defined post-processing functions can be applied to the generated document to modify its rendered appearance.
    5. Writing to file: The final result is saved to $(outputdir)/$(name) with the appropriate extension (.md, .ipynb, or .jl).
  6. How document generation differs by output target

    master

    The behavior of the Document generation step depends on the chosen output format:

    Output TargetMarkdown ChunksCode Chunks
    MarkdownPrinted as-isWrapped in code fences (defaults to @example-blocks)
    NotebookRendered in markdown cellsRendered in code cells
    ScriptDiscardedPrinted as-is
  7. Understand the Literate source file format

    master

    Literate source files are standard Julia (.jl) scripts. This design allows scripts to serve as both documentation and executable code that can be included in test suites (e.g., via include) to ensure examples remain up to date.

    Basic Syntax Rules:

    • Lines starting with # (hash followed by a space) are treated as Markdown.
    • All other lines are treated as Julia code.
    • Leading whitespace before # is allowed but is removed during output generation.
    • To use regular Julia comments that render as # in the output, use ## (double hash).
    ## This renders as a Julia comment in output
    # This is a Markdown heading
    #
    # This is a Markdown paragraph.
    
    x = 1 // 2
  8. Manually control chunk splits using chunk-splitters

    master

    By default, Literate groups adjacent lines of the same type into chunks. To manually force a split between chunks (for example, to separate code into different notebook cells or @example blocks), use chunk-splitter tokens at the start of a line:

    • #-: Forces a split. The rest of the line is discarded. You can use decorative versions like #------------- for readability.
    • #+: Forces a split and enables Documenter.jl "continued"-blocks.

    Example of forcing a split in a Julia code block:

    x = 1 // 3
    y = 2 // 5
    #-
    z = x + y
  9. Implement custom parsing for compatible admonitions

    master

    Since admonitions are not natively supported in Jupyter notebooks, you can use a preprocess function in Literate.markdown or Literate.notebook to transform a custom syntax into different formats.

    For example, you can use a custom tag like #note # and write a preprocessor that converts it into a standard !!! note admonition for Markdown files, and a blockquote (> *Note*) for Notebook files.

    function md_note(str)
        str = replace(str, r"^#note # (.*)$"m => s"""
        # !!! note
        #     \1""")
        return str
    end
    
    function nb_note(str)
        str = replace(str, r"^#note # (.*)$"m => s"""
        # > *Note*
        # > \1""")
        return str
    end
    
    using Literate
    
    # For Markdown
    Literate.markdown("example.jl", "tmp/"; preprocess = md_note)
    
    # For Notebooks
    Literate.notebook("example.jl", "tmp/"; preprocess = nb_note)
  10. Use multiline comments and markdown strings

    master

    Literate supports two ways to write large blocks of Markdown without prefixing every line with # :

    1. Julia Multiline Comments

    Use standard Julia multiline comments (#= ... =#). The start and end tokens must be on their own lines. Literate rewrites these to regular # comments during preprocessing.

    2. Literal Markdown Strings

    You can use md""" ... """ to define markdown blocks. Note: This feature is not enabled by default. You must pass mdstrings=true to Literate.markdown, Literate.notebook, or Literate.script to use it.

    #= 
    This multiline comment
    is treated as markdown.
    =#
    
    # Enabling md""" strings:
    Literate.markdown(io, "file.jl"; mdstrings=true)
    
    md"""
    # Title
    blah blah blah
    """
  11. Configure output directory and filename

    master

    The final step of the pipeline writes the generated file to a specific location:

    • Path structure: $(outputdir)/$(name)(.md|.ipynb|.jl)
    • outputdir: The directory supplied by the user (e.g., docs/generated).
    • name: The filename supplied by the user.

    Best Practice: Add your outputdir to .gitignore, as these files are intended to be generated as part of the build process rather than being tracked in version control.

  12. Integrate Literate.jl with Documenter.jl

    master

    When using Literate to generate documentation for a Julia package, you can pass the keyword argument documenter = true to the generators (Literate.markdown, Literate.notebook, or Literate.script). This enables specific transformations that make your source files compatible with Documenter.jl features while ensuring the output files (Markdown, Notebooks, or Scripts) are correctly formatted for their respective environments.

    # Example usage pattern
    Literate.markdown(["src/example.jl"], "docs/src"; documenter = true)