DOCSEARCHbyAlgolia
Submit Docs

Together Cookbook

repository·main·Indexed 22 days ago

https://github.com/togethercomputer/together-cookbook

A collection of practical code recipes and guides for implementing open source models using Together AI's infrastructure. It includes tutorials on vision-language capabilities with Qwen3.5, agentic workflows using frameworks like LangGraph and PydanticAI, LLM evaluations, and fine-tuning methodologies including LoRA and DPO. Additionally, it provides implementations for the Code Interpreter OpenEnv environment and GRPO training for BlackJack on Together Instant Clusters.

Tokens
21.3K
Snippets
64
Records
89
Agent score
77%

What's inside Together Cookbook

  1. Overview of the Together Cookbook

    main
    The Together Cookbook is a repository of code snippets and guides intended to help developers build applications using open source models via the Together AI platform. The primary use case is to copy and integrate these recipes directly into your own projects.

  2. Access Together AI developer resources

    main

    For technical implementation, API references, and community support, use the following official Together AI resources:

    • Developer Documentation: Comprehensive guides for working with open source models.
    • API Support Docs: Detailed reference for API endpoints, including Chat Completions.
    • Discord Community: For real-time support and community interaction.
    • Research & Blog: For technical papers, research findings, and product announcements.

  3. Understand image token calculation for Together AI

    main

    When using vision models on Together AI, image tokens are calculated based on the image dimensions (Height H and Width W). This affects your total token count and cost.

    The formula used is: T = min(2, max(H // 560, 1)) * min(2, max(W // 560, 1)) * 1601

    This typically results in approximately 1,601 to 6,404 tokens per image.

  4. How the Code Interpreter OpenEnv integration works

    main
    This integration wraps Together's Code Interpreter into the OpenEnv framework. It uses a server/client architecture where a server (running code_interpreter_env.server.app) wraps the Code Interpreter functionality and exposes it via an HTTP interface. The client interacts with this server using the standard OpenEnv API (reset, step, close), allowing the Code Interpreter to be treated as a modular, swappable environment in reinforcement learning or training pipelines alongside traditional games.

  5. Understand the Qwen3-VL Coordinate System

    main

    Qwen3-VL uses a relative coordinate system scaled from 0 to 1000. When performing grounding or detection tasks, coordinates are expressed in this range:

    • bbox_2d: [x1, y1, x2, y2] (top-left and bottom-right corners)
    • point_2d: [x, y] (point coordinates)
    • bbox_3d: [x, y, z, x_size, y_size, z_size, roll, pitch, yaw] (3D bounding box parameters)

  7. Implement Retrieval-Augmented Generation (RAG)

    main

    The cookbook provides several RAG implementation strategies:

    • Reasoning RAG: Combining RAG with reasoning models (e.g., DeepSeek R1) for source citations.
    • Multimodal RAG: Using images (e.g., Nvidia slide decks) as retrieval context.
    • Contextual RAG: Implementing 'Contextual Retrieval' using open models or deploying via Union.ai.
    • Standard RAG: Text-based retrieval-augmented generation workflows.

  8. Implement LLM Evaluations (Evals)

    main

    Use the following patterns to evaluate model performance and prompt effectiveness:

    • Classification Evals: Use 'LLM-as-a-Judge' for safety and classification tasks.
    • Comparison Evals: Perform head-to-head model comparisons (e.g., on summarization tasks).
    • Prompt Evals: Optimize prompts through A/B testing.
    • Judge Optimization: Tune LLM-as-judge configurations to better align with human judgment.
    • Genetic Optimization (GEPA): Optimize prompts via genetic prompt evolution on Together models.

  9. Deploy Kubernetes manifests for GRPO BlackJack

    main

    Deploy the necessary infrastructure (PVC, workspace pod, and BlackJack server) using the provided manifests.

    Important: Before applying, check k8s-manifests.yaml and ensure the volumeName matches the PersistentVolume/PVC available in your cluster. The default is set to together-openenv-integration.

    cd together-cookbook/OpenEnv_GRPO_BlackJack
kubectl apply -f k8s-manifests.yaml

  10. Launch a fine-tuning job via the Together API

    main

    You can initiate a fine-tuning job using the client.fine_tuning.create method. This cookbook uses a helper function send_ft_job to wrap this logic.

    Key parameters for client.fine_tuning.create include:

    • training_file: The ID of the uploaded file to use for training.
    • model: The base model to fine-tune (e.g., meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference).
    • n_epochs: Number of training epochs.
    • learning_rate: The learning rate for the optimizer.
    • lora: Boolean to enable LoRA fine-tuning.
    • lora_r, lora_alpha, lora_dropout: LoRA-specific hyperparameters.
    • train_on_inputs: Whether to train on the input text as well as the output.
    • suffix: A custom suffix for the fine-tuned model name.
    def send_ft_job(client,
                model="meta-llama/Meta-Llama-3.1-8B-32k-Instruct-Reference",
                n_epochs=4,
                run_name='1113-summarization-long-context-finetune',
                train_on_inputs=False,
                learning_rate=6e-5,
                filename=None,
                summarization_file_id=None):
    if filename:
        response = client.files.upload(filename, check=True)
        summarization_file_id = response.id
    else:
        assert summarization_file_id is not None, "provide summarization_file_id"

    response = client.fine_tuning.create(
        training_file = summarization_file_id,
        model = model,
        n_epochs = n_epochs,
        n_checkpoints = 1,
        batch_size = "max",
        learning_rate = learning_rate,
        warmup_ratio = 0.05,
        suffix=run_name,
        wandb_api_key = WANDB_API_KEY,
        lora=True,
        lora_r=32,
        lora_alpha=64,
        lora_dropout=0.05,
        train_on_inputs=train_on_inputs,
    )
    return response.id

  12. Run the Together Code Interpreter OpenEnv demo

    main

    The demo requires a server-client architecture. You must run the server in one terminal and the client/demo in another. Ensure you have set your TOGETHER_API_KEY environment variable in both terminals.

    1. Terminal 1: Start the Code Interpreter OpenEnv server.
    2. Terminal 2: Run the demo script which uses the OpenEnv client.
    # Terminal 1: Start Code Interpreter OpenEnv server
export TOGETHER_API_KEY="your-api-key"
python -m code_interpreter_env.server.app

# Terminal 2: Run the demo (uses OpenEnv client)
export TOGETHER_API_KEY="your-api-key"
python code_interpreter_demo.py