DATAGEN Documentation

DATAGEN uses a three-level loading strategy to optimize the Context Window, loading more information only when necessary:

1. Level 1: Metadata (Loaded at system startup)

   • Contains only name and description.
   • Very lightweight (~100 tokens).

2. Level 2: Instructions (Loaded when agent is triggered)

   • Full system prompt from AGENT.md content.
   • Auto-injected global rules.

3. Level 3: Resources (Loaded when agent calls lookup_skill)

   • Full SKILL.md content.
   • MCP server resources.
   • External files.

DATAGEN uses LangGraph to manage a state graph that coordinates several specialized agents. The typical research workflow follows these steps:

1. Hypothesis Generation: The hypothesis_agent generates research hypotheses.
2. Human Intervention: The user chooses whether to continue with the hypothesis or regenerate it.
3. Processing: A coordinated effort involving:
   • code_agent: Writes data analysis code.
   • process_agent: Supervises the research process.
   • searcher_agent: Conducts web/literature searches.
   • visualization_agent: Creates data visualizations.
   • note_agent: Records the research process for state tracking.
4. Quality Review: The quality_review_agent performs a review.
5. Revision: The system revises the output as needed based on the review.

All skills must be stored within the config/skills/ directory. Each skill requires its own subdirectory named after the skill, containing a mandatory SKILL.md file.

Standard Structure:

config/skills/
└── {skill-name}/
    └── SKILL.md       # Skill definition file (required)

Multi-File Structure: For complex skills, you can include additional files like documentation or scripts within the same directory:

config/skills/advanced-skill/
├── SKILL.md           # Main instruction file
├── REFERENCE.md       # Detailed reference
└── scripts/
    └── helper.py      # Helper scripts

You can reference these files in your SKILL.md using standard Markdown links, e.g., [REFERENCE.md](REFERENCE.md).

Skills are reusable knowledge modules that provide agents with domain-specific expertise. To optimize context window usage, DATAGEN uses a progressive disclosure mechanism:

1. Level 1 (System Startup): The agent is only aware of the skill's name and description.
2. Level 2 (On Demand): The agent retrieves the full content of the skill by calling lookup_skill("{skill-name}") only when the specific expertise is required.

This approach prevents unnecessary context window consumption by not loading full skill instructions until they are actually needed for a task.

To extend the agent's capabilities with a new tool, follow these three steps:

1. Create the tool function in src/tools/.
2. Register the tool in src/tools/factory.py within the ToolFactory._registry dictionary.
3. Add the tool name to the agent's config.yaml under the tools list.

# 2. Register in src/tools/factory.py
from .my_tools import my_custom_tool

class ToolFactory:
    _registry = {
        # ... existing tools ...
        "my_custom_tool": my_custom_tool,
    }

Each agent in DATAGEN requires a dedicated directory within config/agents/{agent_name}/. This directory must contain two mandatory files:

1. AGENT.md: Contains the system prompt and metadata.
2. config.yaml: Defines the agent's capabilities, tools, and skills.

Directory Structure:

config/agents/{agent_name}/
├── AGENT.md       # System prompt (required)
└── config.yaml    # Capabilities config (required)

To install DATAGEN, you need Python 3.10 or higher. Follow these steps to set up your environment using Conda:

1. Clone the repository:

   git clone https://github.com/starpig1129/DATAGEN.git

2. Create and activate a Conda virtual environment:

   conda create -n datagen python=3.10
   conda activate datagen

3. Install dependencies:

   pip install -r requirements.txt

4. Set up environment variables by renaming .env Example to .env and filling in the required values.

To make a skill available to an agent, add its name to the skills list in the agent's config.yaml file.

skills:
  - data-validation

To enforce rules across the entire agent system (such as output formatting or security guidelines), edit the shared rules file at config/agents/_shared/rules.md. Every agent will automatically incorporate these rules into their operation.

# Global Rules

## Output Format
- All code must include type hints
- Use Google Style docstrings

## Security Guidelines
- Do not execute file deletion operations
- Sensitive data must be anonymized