DOCSEARCHbyAlgolia
Submit Docs

StarVector

repository·main·Indexed 26 days ago

https://github.com/joanrod/star-vector

A multimodal vision-language model designed for generating Scalable Vector Graphics (SVG) from images and text. StarVector treats SVG generation as a code generation task and is optimized for icons, logotypes, technical diagrams, graphs, and charts. The library supports image-to-SVG and text-to-SVG tasks, providing pretrained models such as StarVector-1B and StarVector-8B, and integrates with HuggingFace and vLLM backends.

Tokens
7.3K
Snippets
20
Records
40
Agent score
88%

What's inside StarVector

  3. Finetune StarVector (Text2SVG and SVG-Bench)

    main

    After pretraining, you can finetune the models on specific tasks.

    Text2SVG Finetuning:

    • StarVector-1B: Use accelerate launch with configs/models/starvector-1b/text2svg-stack.yaml.
    • StarVector-8B: Use torchrun with configs/models/starvector-8b/text2svg-stack.yaml.

    SVG-Bench Finetuning:

    • StarVector-1B: Use accelerate launch with configs/models/starvector-1b/im2svg-{fonts,icons,emoji}.yaml.
    • StarVector-8B: Use torchrun with configs/models/starvector-8b/im2svg-{fonts,icons,emoji}.yaml.

  4. Access StarVector via Jupyter Notebook

    main

    Once the Docker container is running, Jupyter Notebook will be available on port 8888. Look for the URL in the container logs, which will follow this format:

    http://127.0.0.1:8888/?token=<your_token_here>

    Copy and paste this URL into your web browser to access the notebook environment.

  5. Run the StarVector Docker Container

    main

    Run the StarVector container with GPU support, Jupyter Notebook exposure, and volume mounting for project files and Hugging Face cache. You must provide a HUGGING_FACE_HUB_TOKEN to access gated models like bigcode/starcoderbase-1b.

    docker run -it \
  --gpus all \
  -p 8888:8888 \
  -v $(pwd)/..:/workspace \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  --env HUGGING_FACE_HUB_TOKEN=<your_huggingface_token> \
  --name starvector \
  starvector:latest

  6. Run StarVector Demo with vLLM Backend

    main

    For faster generation, use the vLLM backend. This requires the StarVector fork of vLLM.

    1. Install StarVector vLLM fork:

      git clone https://github.com/starvector/vllm.git
cd vllm
pip install -e .

    2. Launch the vLLM endpoint:

      vllm serve starvector/starvector-1b-im2svg --chat-template configs/chat-template.jinja --trust-remote-code --port 8000 --max-model-len 8192

    3. Launch the Gradio Controller:

      python -m starvector.serve.vllm_api_gradio.controller --host 0.0.0.0 --port 10000

    4. Launch the Gradio Web Server:

      python -m starvector.serve.vllm_api_gradio.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --port 7000

    5. Launch the Model Worker:

      python -m starvector.serve.vllm_api_gradio.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-name starvector/starvector-1b-im2svg --vllm-base-url http://localhost:8000
    # 1. Launch the VLLM endpoint
vllm serve starvector/starvector-1b-im2svg --chat-template configs/chat-template.jinja --trust-remote-code --port 8000 --max-model-len 8192

# 2. Create the demo for VLLM
python -m starvector.serve.vllm_api_gradio.controller --host 0.0.0.0 --port 10000
python -m starvector.serve.vllm_api_gradio.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --port 7000
python -m starvector.serve.vllm_api_gradio.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-name starvector/starvector-1b-im2svg --vllm-base-url http://localhost:8000

  8. Run StarVector Demo with Gradio Web UI (HuggingFace Backend)

    main

    To use the Gradio web interface with the HuggingFace generation backend, follow these three steps in order:

    1. Launch the Controller:

      python -m starvector.serve.controller --host 0.0.0.0 --port 10000

    2. Launch the Gradio Web Server:

      python -m starvector.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --port 7000

    3. Launch a Model Worker: (This performs the actual GPU inference)

      python -m starvector.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path joanrodai/starvector-1.4b

    Note: You can launch multiple workers by changing the --port and --worker arguments while keeping the same --controller URL.

    # Launch a controller
python -m starvector.serve.controller --host 0.0.0.0 --port 10000

# Launch a gradio web server
python -m starvector.serve.gradio_web_server --controller http://localhost:10000 --model-list-mode reload --port 7000

# Launch a model worker
python -m starvector.serve.model_worker --host 0.0.0.0 --controller http://localhost:10000 --port 40000 --worker http://localhost:40000 --model-path joanrodai/starvector-1.4b

  9. Quick Start: Image-to-SVG Generation with StarVectorForCausalLM

    main

    Use the StarVectorForCausalLM class to perform image-to-SVG vectorization. This method involves loading a pretrained model, processing an input image, and generating the SVG code.

    from PIL import Image
from starvector.model.starvector_arch import StarVectorForCausalLM
from starvector.data.util import process_and_rasterize_svg

model_name = "starvector/starvector-8b-im2svg"

starvector = StarVectorForCausalLM.from_pretrained(model_name)

starvector.cuda()
starvector.eval()

image_pil = Image.open('assets/examples/sample-0.png')
image = starvector.process_images([image_pil])[0].cuda()
batch = {"image": image}

raw_svg = starvector.generate_im2svg(batch, max_length=1000)[0]
svg, raster_image = process_and_rasterize_svg(raw_svg)

  10. Run StarVector validation using HuggingFace backend

    main

    To evaluate StarVector models using the HuggingFace generation API, run the validate.py script with the appropriate configuration file and dataset name. This supports both StarVector-1B and StarVector-8B models.

    # StarVector-1B
python starvector/validation/validate.py \
config=configs/generation/hf/starvector-1b/im2svg.yaml \
dataset.name=starvector/svg-stack

# StarVector-8B 
python starvector/validation/validate.py \
config=configs/generation/hf/starvector-8b/im2svg.yaml \
dataset.name=starvector/svg-stack

  11. Install StarVector

    main

    To install StarVector, clone the repository, create a Conda environment with Python 3.11.3, and install the package in editable mode. For training capabilities, install the [train] extra.

    # 1. Clone and navigate
git clone https://github.com/joanrod/star-vector.git
cd star-vector

# 2. Install Package
conda create -n starvector python=3.11.3 -y
conda activate starvector
pip install --upgrade pip
pip install -e .

# 3. Install additional packages for training
pip install -e ".[train]"

  12. Train StarVector-1B (Image2SVG Pretraining)

    main

    Train StarVector-1B on the SVG-Stack dataset for the Image2SVG task using Deepspeed and Accelerate. Ensure you are in the star-vector root directory.

    # StarVector-1B
accelerate launch --config_file configs/accelerate/deepspeed-8-gpu.yaml starvector/train/train.py config=configs/models/starvector-1b/im2svg-stack.yaml