DOCSEARCHbyAlgolia
Submit Docs

Easy Dataset

repository·main·Indexed 12 days ago

https://github.com/conardli/easy-dataset

An application for automating the creation of high-quality fine-tuning and evaluation datasets for Large Language Models (LLMs). It transforms unstructured documents (PDF, Markdown, DOCX, TXT, EPUB) into structured formats such as Single-Turn QA, Multi-Turn Dialogue, and Image QA. Version 1.7.3 supports OpenAI-format APIs, including Ollama, MiniMax, and Zhipu AI, and exports to Alpaca, ShareGPT, and Multilingual-Thinking formats.

Tokens
13.7K
Snippets
42
Records
66
Agent score
96%

What's inside Easy Dataset

  1. Overview of Easy Dataset

    main
    Easy Dataset is a powerful tool designed for creating fine-tuning datasets for Large Language Models (LLMs). It provides an intuitive interface alongside built-in document parsing, intelligent segmentation, data cleaning, and enrichment capabilities. The application can transform domain-specific documents in various formats into high-quality structured datasets suitable for scenarios like model fine-tuning, Retrieval-Augmented Generation (RAG), and model performance evaluation.

  2. Overview of Easy Dataset features

    main

    Easy Dataset is a tool for building Large Language Model (LLM) datasets. It provides a suite of capabilities for the entire data pipeline:

    • Document Processing: Intelligent parsing of PDF, Markdown, DOCX, TXT, and EPUB. Includes text splitting (Markdown structure, recursive, fixed length, code-aware) and data cleaning.
    • Data Generation: Automated question generation from text, answer generation with Chain of Thought (COT), and domain label tree construction.
    • Dataset Types: Supports Single-Turn QA, Multi-Turn Dialogue, Image QA, and Data Distillation (generating data from topics without documents).
    • Model Evaluation: Automated evaluation using a Judge Model, human blind testing (Arena), and generation of various test formats (true/false, multiple-choice, etc.).
    • Export & Integration: Exports to Alpaca, ShareGPT, and Multilingual-Thinking formats (JSON/JSONL). Supports LLaMA Factory configuration and direct Hugging Face Hub uploads.
    • Model Support: Compatible with any OpenAI-format API, including OpenAI, Ollama (local), MiniMax, Zhipu AI, Alibaba Bailian, and OpenRouter. Supports vision models like Gemini and Claude.

  3. Understand the Easy DataSet Architecture

    main

    Easy DataSet is an application designed to create fine-tuning datasets for Large Language Models (LLMs). The workflow involves uploading text files, automatically splitting them into segments, generating questions via LLM APIs, and finally producing a complete dataset for fine-tuning.

    Core Tech Stack:

    • Frontend: Next.js 14 (App Router) with Material-UI (MUI)
    • Storage: Local file system (fs) acting as a simulated database
    • Language: JavaScript
    • Internationalization: i18next supporting English (en) and Simplified Chinese (zh-CN).

  4. Understand the Easy DataSet Project Architecture

    main

    Easy DataSet is an application designed to create fine-tuning datasets for Large Language Models (LLMs). The workflow involves uploading text files, automatically splitting text into chunks, generating questions from those chunks, and finally producing datasets for model fine-tuning.

    Core Workflow

    1. Project Creation: Users define a project with a name and description. The system initializes a project directory and configuration.
    2. Document Processing: Users upload Markdown files. The system saves the original file, splits the text into manageable chunks, and generates a table of contents.
    3. Question Generation: Users select text chunks. The system calls an LLM API to generate questions, which are then stored in a question list and a label tree.
    4. Dataset Creation: Users select questions for answer generation. The system calls an LLM API to generate answers, saves the resulting dataset, and provides export functionality.

  5. Internationalization (i18n) Setup

    main

    The project uses i18next and react-i18next for multi-language support.

    Supported Languages:

    • English (en)
    • Simplified Chinese (zh-CN)
    • Turkish (tr)

    Implementation Details:

    • Translations are stored in locales/{lang}/translation.json.
    • Configuration is handled via lib/i18n.js.
    • Language detection is managed by i18next-browser-languagedetector.

  6. Understand the Project Directory Structure

    main

    The project follows a Next.js 14 (App Router) structure:

    • app/: Contains Next.js application routes, including api/ for backend logic and projects/[projectId]/ for project-specific views.
    • components/: Contains React UI components (e.g., Navbar.js, CreateProjectDialog.js).
    • lib/: The utility library, specifically lib/db/ for data management.
    • locales/: Internationalization (i18n) translation files for English (en), Simplified Chinese (zh-CN), and Turkish (tr).
    • local-db/: The directory where the simulated file-system database resides.

  7. Explore the Project Directory Structure

    main

    The project follows a standard Next.js App Router structure with a dedicated library for data management:

    • app/: Contains Next.js routes, including API routes (api/) and project-specific pages (projects/[projectId]/).
    • components/: React components for the UI (e.g., Navbar.js, CreateProjectDialog.js).
    • lib/db/: The core data management layer. It uses the file system to manage projects, texts, and datasets.
    • local-db/: The directory where the simulated database (JSON files) is stored.
    • locales/: Internationalization resource files for en, zh-CN, and pt-BR.
    easy-dataset/
├── app/                      # Next.js application directory
│   ├── api/                 # API routes
│   │   └── projects/       # Project-related APIs
│   ├── projects/           # Project-related pages
│   │   └── [projectId]/    # Project detail pages
│   └── page.js            # Home page
├── components/             # React components
├── lib/                    # Utility library
│   └── db/                # Database module
│       ├── base.js        # Base utilities
│       ├── projects.js    # Project management
│       ├── texts.js       # Text processing
│       ├── datasets.js    # Dataset management
│       └── index.js       # Module exports
├── local-db/             # Local database directory
└── locales/              # i18n resource directory

  9. How the Data Flow Works

    main

    The application operates through four primary workflows:

    1. Project Creation: Users create a project via the home page or navbar. The system initializes the project directory and configuration files, then redirects to the project detail page.
    2. Document Processing: Users upload Markdown files. The system saves the original file, calls a text-splitting service to generate segments and a directory structure, and displays the results.
    3. Question Generation: Users select specific text segments. The system calls an LLM API to generate questions, which are then saved to a question list and a tag tree.
    4. Dataset Generation: Users select questions to generate answers. The system calls an LLM API to produce answers, saves the dataset, and provides an export option.

  10. How the Easy Dataset data workflow works

    main

    Easy Dataset follows a linear pipeline to transform raw documents into structured LLM fine-tuning datasets:

    1. Document Upload: Loading files into the system.
    2. Text Splitting: Using smart chunking to break documents into manageable pieces.
    3. Question Generation: Using an LLM to generate questions based on text chunks.
    4. Answer Generation: Using an LLM to generate answers for those questions.
    5. Dataset Export: Converting the generated pairs into the desired output format.

  11. Build and Run with a Local Dockerfile

    main

    If you prefer to build the Docker image locally from the project root:

    1. Clone the repository and enter the directory.
    2. Build the image:
      docker build -t easy-dataset .
    3. Run the container:
      docker run -d \
  -p 1717:1717 \
  -v ./local-db:/app/local-db \
  -v ./prisma:/app/prisma \
  --name easy-dataset \
  easy-dataset

    Note: Mounting ./local-db and ./prisma is recommended for database consistency. The database initializes automatically on startup.

    docker build -t easy-dataset .
docker run -d \
  -p 1717:1717 \
  -v ./local-db:/app/local-db \
  -v ./prisma:/app/prisma \
  --name easy-dataset \
  easy-dataset

  12. Install Easy Dataset via NPM

    main

    To run Easy Dataset in a development environment using NPM, follow these steps:

    1. Clone the repository:
      git clone https://github.com/ConardLi/easy-dataset.git
cd easy-dataset
    2. Install dependencies:
      npm install
    3. Build and start the development server:
      npm run build
npm run start
    4. Access the application at http://localhost:1717.
    git clone https://github.com/ConardLi/easy-dataset.git
cd easy-dataset
npm install
npm run build
npm run start