DeTikZify

repository·main·Indexed 23 days ago

https://github.com/potamides/detikzify

A multimodal language model designed to automatically synthesize scientific figures and sketches as semantics-preserving TikZ graphics programs. It features an MCTS-based inference algorithm for iterative refinement, a Web UI for interactive synthesis, and the DetikzifyPipeline for programmatic image-to-TikZ and text-to-TikZ (via TikZero+) generation.

Tokens
3.8K
Snippets
11
Records
19
Agent score
77%

What's inside detikzify

  1. Reproduce DeTi*k*Zify v1 vs v2

    main
    The current scripts in the examples/ directory are designed for the DeTikZify<sub>v2</sub> training pipeline and model state. If you need to reproduce results for DeTikZify<sub>v1</sub>, you must switch to a previous release of the repository via GitHub releases.
  2. Start the DeTi*k*Zify Web UI

    main

    For interactive use and general usage tips, you can start the DeTikZify Web UI directly from the command line. Use the --light flag to launch it.

    To see all available options, run the command with --help.

    python -m detikzify.webui --light
  3. Use DeTi*k*Zify example scripts for training or evaluation

    main
    The scripts in the examples/ directory can be used to train or pretrain your own DeTikZify models, or to reproduce the project's evaluation results. Each script provides a command-line interface (CLI). To see the available options and parameters for a specific script, run it with the --help flag.
  4. Explore Ti*k*Zero adapter training examples

    main
    The examples/tikzero/ directory contains scripts specifically designed for training TikZero adapters. Each script provides a command-line interface (CLI) for training tasks. To see the available configuration options and arguments for a specific training script, run the script with the --help flag.
  5. Install DeTi*k*Zify via pip

    main

    You can install the DeTikZify Python package using pip. Use the [legacy] extra if you intend to use DeTikZify v1 models; otherwise, it can be omitted for v2 models.

    Note: DeTikZify requires a full TeX Live 2023 installation, ghostscript, and poppler to be installed on your system via your package manager.

    pip install 'detikzify[legacy] @ git+https://github.com/potamides/DeTikZify'
  6. Implement Monte Carlo Tree Search (MCTS)

    main

    To use the MCTS library, you must provide a MonteCarlo instance initialized with a root Node. Depending on your approach (Traditional vs. Expert Policy), you need to define how the tree expands and how nodes are evaluated.

    Core Requirements

    1. child_finder: A function that finds the direct children of each search tree node and adds them to the parent.
    2. node_evaluator (Traditional only): A function that evaluates nodes for end-state outcomes (e.g., win, loss, tie).

    Initialization

    from chess import Game
    from montecarlo.node import Node
    from montecarlo.montecarlo import MonteCarlo
    
    chess_game = Game()
    montecarlo = MonteCarlo(Node(chess_game))
  7. Use the DeTi*k*Zify programming interface for image-to-TikZ synthesis

    main

    The DetikzifyPipeline allows you to synthesize TikZ graphics programs from images.

    Key workflow:

    1. Load the model using load().
    2. Initialize DetikzifyPipeline.
    3. Use .sample(image=...) to generate a single TikZ program.
    4. Use .simulate(image=..., timeout=...) to run MCTS-based inference, which returns a generator of (score, fig) tuples.
    5. Use .rasterize().show() on the resulting figure object if is_rasterizable is true to view the result.
    6. Use .save("filename.tex") to save the TikZ code.
    from operator import itemgetter
    from detikzify.model import load
    from detikzify.infer import DetikzifyPipeline
    
    image = "https://w.wiki/A7Cc"
    pipeline = DetikzifyPipeline(*load(
        model_name_or_path="nllg/detikzify-v2.5-8b",
        device_map="auto",
        torch_dtype="bfloat16",
    ))
    
    # generate a single TikZ program
    fig = pipeline.sample(image=image)
    
    # if it compiles, rasterize it and show it
    if fig.is_rasterizable:
        fig.rasterize().show()
    
    # run MCTS for 10 minutes and generate multiple TikZ programs
    figs = set()
    for score, fig in pipeline.simulate(image=image, timeout=600):
        figs.add((score, fig))
    
    # save the best TikZ program
    best = sorted(figs, key=itemgetter(0))[-1][1]
    best.save("fig.tex")
  8. Alternative ways to run DeTi*k*Zify

    main

    If you encounter difficulties with local installation or hardware inference, you can use the following alternatives:

    • Hugging Face Space: Visit the official Hugging Face Space. Note that restarting the space can take up to 30 minutes.
    • Google Colab: Use the official demo on Google Colab. Note that the free tier only supports inference for the 1b models.
    • Docker: You can run the project locally using Docker via the Hugging Face Space options.
    • Duplicate Space: You can duplicate the Hugging Face Space with a paid private GPU runtime.
  9. Install DeTi*k*Zify in editable mode for examples

    main

    If you want to run the included examples, clone the repository and install it in editable mode with the [examples] extra:

    git clone https://github.com/potamides/DeTikZify
    pip install -e DeTikZify[examples]
  10. Tips for improving DeTi*k*Zify synthesis results

    main

    If DeTikZify is struggling to interpret your sketches or figures, consider the following strategies:

    Visual Prompting

    Treat your sketches like prompts for an LLM. If the intent is unclear, simplify the sketch or focus on the key elements. For example, reducing stroke width or using more recognizable characters can significantly improve success rates.

    Image Editor & Input Quality

    • Use External Editors: The integrated editor is limited. For complex figures, draw them in a dedicated graphics editor and upload the file.
    • Minimize Margins: All input images are cropped to the smallest square around their content and resized. To avoid issues like unintended axis thickness, try to fill as much of the canvas as possible.
    • Simplify Complexity: For very complex figures, try segmenting the input into simpler pieces and synthesizing them independently, then assembling them manually.

    Parameter Tuning

    • Accuracy vs. Efficiency: Lowering temperature and top-p (nucleus) values forces the model to follow the input image more closely, but may increase the frequency of LaTeX compilation errors.

    Post-processing & External Graphics

    • Clean Preambles: The model may include unused packages in the LaTeX preamble. It is recommended to review and clean these up.
    • External Graphics: To generate code that uses placeholders for external graphics, draw the example-image.pdf (from the mwe package) in your input sketch.