Synthetic Data Kit

To improve model performance in tool calling (inspired by the Gorilla paper), you can use the cot-enhance type to add detailed API documentation and reasoning to existing conversation examples.

Workflow

1. Prepare base data: Create a JSON file containing basic API usage examples (e.g., simple Q&A pairs without detailed documentation).
2. Enhance: Run the create command with the --type cot-enhance flag. This uses a custom cot_enhancement prompt to inject relevant API syntax, parameter descriptions, and step-by-step reasoning into the assistant's responses.
3. Save: Use the save-as command to export the enhanced dataset in a format like chatml.

Example Configuration

llm:
  provider: "vllm"

vllm:
  api_base: "http://localhost:8000/v1"
  model: "meta-llama/Llama-3.3-70B-Instruct"

generation:
  temperature: 0.3
  num_pairs: 15

curate:
  threshold: 8.0
  batch_size: 10

prompts:
  # Custom rating prompt for API documentation quality
  qa_rating: |\n    Rate these API usage examples on a scale from 1-10 based on:\n    \n    - Documentation quality (0-3): Is the API documentation clear and complete?\n    - API call correctness (0-3): Is the syntax and usage correct?\n    - Explanation quality (0-2): Is the explanation helpful and accurate?\n    - Parameter coverage (0-2): Are parameters well explained and correctly used?\n    \n    Immediately reject any example where the API call doesn't match the documentation.\n    \n    Return valid JSON with ratings:\n    [\n      {"question": "Original question", "answer": "Original answer with documentation", "rating": 8}\n    ]\n    \n    Examples to rate:\n    {pairs}\n    \n  # Custom prompt for enhancing examples with documentation
  cot_enhancement: |\n    You are enhancing API usage examples by adding detailed documentation.\n    \n    For each conversation, add the following to the assistant's responses:\n    1. Relevant API documentation with syntax and parameter descriptions\n    2. Clear explanation of why this API is appropriate\n    3. Description of what each parameter does\n    \n    Return the enhanced conversations as a JSON array matching this format:\n    [\n      {\n        "role": "system", \n        "content": "System message"\n      },\n      {\n        "role": "user", \n        "content": "How do I use function X?"\n      },\n      {\n        "role": "assistant", \n        "content": "Here's the documentation for this API:\n\n```\nfunction X(param1, param2) -> return_type\n  param1: description of param1\n  param2: description of param2\n  returns: what is returned\n```\n\nHere's how you can use it:\n\n```python\nresult = X(value1, value2)\n```\n\nThis works because [detailed explanation]..."\n      }\n    ]\n    \n    Original conversations:\n    {conversations}\nEOF

# Enhance existing examples with documentation using cot-enhance
# 1. First prepare a JSON file with basic API usage examples (without detailed docs)
# Example: api_examples.json with basic Q&A pairs about APIs

# 2. Enhance these examples by adding detailed documentation
synthetic-data-kit -c gorilla_config.yaml create data/api_examples.json --type cot-enhance

# 3. Save the enhanced dataset
synthetic-data-kit -c gorilla_config.yaml save-as data/generated/api_examples_enhanced.json -f chatml

To implement a new generation logic (e.g., Chain-of-Thought):

1. Create the generator class: In the generators/ directory, create a class that accepts an LLMClient and an optional config_path. Implement your generation logic (e.g., generate_cot_examples).
2. Define the prompt: Add the corresponding prompt template to config.yaml under the prompts: key.
3. Integrate with the CLI/Core: Update the process_file logic (or the relevant command handler) to instantiate your new generator and handle the output saving when the specific content_type is requested.

# synthetic_data_kit/generators/cot_generator.py
from typing import Dict, List, Any, Optional
import json
from synthetic_data_kit.models.llm_client import LLMClient
from synthetic_data_kit.utils.config import get_prompt

class COTGenerator:
    """Generates chain-of-thought reasoning examples"""
    
    def __init__(self, client: LLMClient, config_path: Optional[str] = None):
        self.client = client
        self.config = client.config
    
    def generate_cot_examples(self, document_text: str, num_examples: int = 5) -> List[Dict[str, Any]]:
        prompt_template = get_prompt(self.config, "cot_generation")
        prompt = prompt_template.format(num_examples=num_examples, text=document_text)
        
        messages = [{"role": "system", "content": prompt}]
        response = self.client.chat_completion(messages)
        
        examples = []
        if '[' in response and ']' in response:
            start = response.find('[')
            end = response.rfind(']') + 1
            try:
                examples = json.loads(response[start:end])
            except:
                print("Error parsing COT examples")
        
        return examples