SceneSmith

The robot evaluation module automates the process of converting human tasks into generated scenes and validating robot task completion. The pipeline consists of four stages:

1. Generate Prompts: Converts a human task (e.g., "Find a fruit") into diverse scene prompts using an LLM.
2. Generate Scenes: Runs the standard SceneSmith pipeline using the generated prompts.
3. Policy Interface (Optional): Converts scene states into robot-executable poses (target/reference bindings). This is useful for model-based policies.
4. Validate: A VLM agent compares the robot's final output poses against the original scene metadata to verify success.

Robot Policy Contract: The robot must receive the initial scene.dmd.yaml, perform the task, and output a modified scene.dmd.yaml containing the updated object poses.

Choose a backend based on your requirements for quality, speed, and available hardware:

• Hunyuan3D: Diffusion-based image-to-3D. Faster and lower memory footprint.
• SAM3D: Segmentation + Gaussian splatting reconstruction. Higher quality and UV-mapped textures, but slower and requires more VRAM.

To enable PBR material retrieval for scene rendering:

1. Download materials:

   python scripts/download_ambientcg.py --output data/materials

   Options: -r <res> (e.g., 2K), -f <format> (e.g., PNG), --limit <n>, or --dry-run.
2. Download pre-computed CLIP embeddings (Recommended):

   huggingface-cli download nepfaff/scenesmith-preprocessed-data \
    --repo-type dataset \
    --include "ambientcg/embeddings/**" \
    --local-dir data/scenesmith-preprocessed-data
   mv data/scenesmith-preprocessed-data/ambientcg/embeddings data/materials/embeddings
   rm -rf data/scenesmith-preprocessed-data

   *Alternatively, compute them yourself:* `python scripts/compute_ambientcg_embeddings.py --materials-dir data/materials`.

Objaverse (via ObjectThor) provides asset retrieval using GLB meshes and pre-computed CLIP features.

1. Download data (~50GB):

   bash scripts/download_objaverse_data.sh

2. Preprocess for retrieval to create embeddings and metadata:

   python scripts/prepare_objaverse.py

3. Enable Objaverse in your experiment configuration:

   asset_manager:
     general_asset_source: "objaverse"

# In your experiment config file
asset_manager:
  general_asset_source: "objaverse"  # Use Objaverse retrieval

Generated output directories can become very large. Use the provided script to remove all files that are not required for the final scene.

python scripts/clean_scene_output.py

You can run the server as an independent microservice or for testing/debugging using the standalone_server module. By default, it uses the Hunyuan3D backend. To use the SAM3D backend, you must provide the required checkpoint paths via CLI flags.

# Hunyuan3D backend (default)
python -m scenesmith.agent_utils.geometry_generation_server.standalone_server

# SAM3D backend
python -m scenesmith.agent_utils.geometry_generation_server.standalone_server \
  --backend sam3d \
  --sam3-checkpoint external/checkpoints/sam3.pt \
  --sam3d-checkpoint external/checkpoints/pipeline.yaml

Use pytest to run unit or integration tests. The --testmon flag enables test monitoring/caching. Use -x to stop on the first failure.

# Run unit tests
pytest tests/unit/ --testmon

# Run integration tests
pytest tests/integration/ --testmon

# Run all tests
pytest tests/ --testmon

SceneSmith uses uv for dependency management. Follow these steps to set up a local development environment:

1. Install uv:

   curl -LsSf https://astral.sh/uv/install.sh | sh

2. Sync dependencies (creates .venv):

   uv sync

   To exclude development tools (like pytest), use uv sync --no-dev.
3. Activate the environment:

   source .venv/bin/activate

4. Install pre-commit hooks:

   pre-commit install

Note: Installation is not complete after these steps. You must still configure asset generation backends (SAM3D or Hunyuan3D-2), articulated object datasets, and PBR materials.