DOCSEARCHbyAlgolia
Submit Docs

Bespoke Curator

repository·main·Indexed 23 days ago

https://github.com/bespokelabsai/curator

A high-performance Python library for bulk inference and scalable data curation, optimized for post-training tasks such as fine-tuning and distillation. It includes features for agent-based multi-turn conversations, code execution for dataset generation (e.g., charts and math animations), multimodal extraction from PDFs using Gemini models, and support for RAFT (Retrieval Augmented Fine-Tuning) workflows.

Tokens
14.8K
Snippets
42
Records
96
Agent score
80%

What's inside bespokelabs-curator

  1. Overview of Bespoke Curator

    main

    Bespoke Curator is a Python library designed for creating scalable synthetic data pipelines. It is used for generating and curating high-quality data for model training (finetuning/distillation) or structured data extraction.

    Key features include:

    • Synthetic Data Generation: Rich Python-based library for generating and curating data.
    • Monitoring: A built-in viewer to monitor data generation in real-time.
    • Structured Outputs: First-class support for ensuring LLM outputs follow specific schemas.
    • Performance & Reliability: Built-in optimizations for asynchronous operations, caching, and fault recovery.
    • Flexible Inference: Supports various inference backends via LiteLLM, vLLM, and popular batch APIs (including OpenAI, Anthropic, and Gemini).

  2. Explore curator usage examples

    main

    The examples/ directory contains several recipes for different data generation and annotation tasks:

    • Persona Hub: Using diverse personas from persona-hub to generate diverse datasets.
    • Ungrounded QA: Generating diverse question-answer pairs using techniques similar to the Camel paper.
    • Poem Generation: Generating diverse poems.
    • Recipe Generation: Using curator with the litellm backend to generate recipes using non-OpenAI models.
    • Reannotation: Taking an existing dataset and reannotating it with a new model.
    • vLLM Online Generation: Setting up a vLLM OpenAI-compatible server to serve a local model for online generation.
    • vLLM Offline Inference: Running curator with a local model offline via vLLM.

  3. Overview of the Math Animation pipeline components

    main

    The math animation example is composed of four main scripts that handle the lifecycle of animation generation:

    • script_generator.py: Generates mathematical concepts and detailed outlines.
    • generate_script.py: Creates hierarchical math content including subjects, topics, and questions.
    • generate_manim_code.py: Converts math concepts into executable Manim code.
    • execute_code.py: Runs the Manim code within a Docker container (with Manim installed) and extracts the generated videos.

  4. Understand the Agent Multi-Turn code structure

    main

    The multi-turn agent framework is built around several key components:

    • Agent Classes (Patient and Doctor): Concrete implementations of agents that represent specific roles.
    • System Prompts: Used to define the role and behavior of each agent.
    • MultiTurnAgents: A handler class that manages the conversation flow between the agents.
    • Conversation History: The system tracks and prints the complete history of the interaction.

  5. Enable Batch Mode to reduce LLM costs

    main
    Many providers (OpenAI, Anthropic, etc.) offer significant discounts for using their batch APIs. In Curator, you can enable this by setting batch=True when initializing your curator.LLM instance. This is particularly useful for large-scale data generation or inference tasks.

  6. Extend curator.LLM for custom prompt and parse logic

    main

    For complex workflows where you need to map inputs to specific prompts or transform structured responses into custom dictionary formats, you can subclass curator.LLM.

    To implement a custom class, override:

    1. response_format: The Pydantic model for the output.
    2. prompt(self, input: Dict) -> str: Defines how to construct the prompt from the input data.
    3. parse(self, input: Dict, response: YourPydanticModel) -> List[Dict]: Defines how to transform the LLM response and the original input into a list of dictionaries (compatible with HuggingFace Datasets).
    from typing import Dict, List
from datasets import Dataset
from pydantic import BaseModel, Field
from bespokelabs import curator

class Poem(BaseModel):
    poem: str = Field(description="一首诗。")

class Poems(BaseModel):
    poems: List[Poem] = Field(description="诗歌列表。")

class Poet(curator.LLM):
    response_format = Poems

    def prompt(self, input: Dict) -> str:
        return f"写两首关于{input['topic']}的诗。"

    def parse(self, input: Dict, response: Poems) -> Dict:
        return [{"topic": input["topic"], "poem": p.poem} for p in response.poems]

poet = Poet(model_name="gpt-4o-mini")
topics = Dataset.from_dict({"topic": ["繁华都市中的都市孤独", "Bespoke Labs 的 Curator 库的美"]})
poem = poet(topics)
print(poem.to_pandas())

  7. Implement a custom CodeExecutor

    main

    To perform custom logic during data processing, you can extend the CodeExecutor base class. You must implement three required methods:

    1. code(): Returns the Python code string to be executed.
    2. code_input(): Defines the input variables/data required by the code.
    3. code_output(): Defines how the execution output should be captured and mapped back to the dataset.

    In the provided example, the HelloExecutor class takes a location from a dataset, executes Python code that prints a greeting, and captures that output to add it to the dataset.

  8. Run the Math Animation generation pipeline

    main

    The math animation pipeline consists of three distinct stages: generating mathematical scripts, converting those scripts into Manim code, and executing the code to produce videos. Run these commands in sequence:

    1. Generate Scripts: Creates hierarchical math content (subjects, topics, and questions).
    2. Generate Manim Code: Converts the generated math concepts into executable Manim code.
    3. Execute Code: Runs the Manim code in a Docker container and extracts the resulting videos.

    The final output is a dataset of math animations published to the Hugging Face Hub.

  10. Authenticate with Bespoke Labs for private datasets

    main

    By default, datasets in the Curator Viewer are public. To keep datasets private, track costs, and share with collaborators, associate your session with a Bespoke Labs account using an API key.

    1. Sign up for an account at curator.bespokelabs.ai/auth/signup.
    2. Generate an API key at curator.bespokelabs.ai/home/keys.
    3. Set both BESPOKE_API_KEY and CURATOR_VIEWER environment variables.
    export BESPOKE_API_KEY=<YOUR_API_KEY>
export CURATOR_VIEWER=1

  11. Generate charts using the Curator Code Executor

    main

    You can use the Curator Code Executor to generate visual assets like charts and capture them as part of a dataset. This is achieved by extending the CodeExecutor base class and implementing specific methods to define the code, prepare inputs, and process outputs.

    To implement a custom chart generator, you must implement these three methods in a class extending CodeExecutor:

    • code(): Returns the Python code string to be executed (e.g., using matplotlib).
    • code_input(): Prepares the data required by the code (such as x/y values, titles, or labels).
    • code_output(): Processes the execution results to extract the generated image or other artifacts.