OpenRouter Runner

Before testing models, you must configure your environment variables and install dependencies.

1. Create a .env.dev file in the project root with the following keys:

   • API_URL: The Modal API endpoint URL.
   • RUNNER_API_KEY: Your custom API key.
   • MODEL: The identifier of the model you are testing.

2. Install dependencies using npm or pnpm:

   npm install
   # or
   pnpm install

API_URL=<MODAL_API_ENDPOINT_THAT_WAS_DEPLOYED>
RUNNER_API_KEY=<CUSTOM_KEY_YOU_CREATED_EARLIER>
MODEL=<MODEL_YOU_ADDED_OR_WANT_TO_TEST>

When a model requires unique hardware, libraries, or configurations not covered by existing containers, you must define a new container:

1. Template a Container: Copy an existing container file from runner/containers to use as a starting point.
2. Customize: Modify the class name, image, machine type, and engine. Install any necessary additional libraries.
3. Register the Container:
   • Import the new class in ./containers/__init__.py.
   • Add the model IDs to the appropriate list in that file.
4. Associate with Protocol:
   • Add a ContainerType for your model in modal/shared/protocol.py.
   • Update get_container(model_path: Path, container_type: ContainerType) in modal/runner/containers/__init__.py to define how to build the new container.
5. Download and Deploy: Run modal run runner::download to prepare the models, then deploy.

Before deploying or configuring the OpenRouter Runner, ensure you have the following accounts and tools ready:

1. Modal Account: Required for the primary deployment platform.
2. Hugging Face Account: Required to obtain a token for accessing models and libraries.
3. Poetry: Must be installed on your local machine for dependency management.

To test your models locally, you must first ensure the runner is active using Modal.

1. Navigate to the openrouter-runner/modal directory.
2. Start the runner with the following command:

   modal serve runner

Keep this terminal active while you run test scripts in a separate terminal window.

If the model you want to deploy is compatible with an existing container type (e.g., a VllmContainer_7B), follow these steps:

1. Identify the Model ID: Find the model on Hugging Face (e.g., mistralai/Mistral-7B-Instruct-v0.2).
2. Update the Model List: Add the model ID and its corresponding ContainerType to the DEFAULT_CONTAINER_TYPES dictionary in runner/containers/__init__.py.
3. Handle Permissions: If the model requires gated access (like Llama-2), ensure you have requested access on Hugging Face.
4. Download Models: Run the download command to prepare the model files in storage.
5. Test: Proceed to testing the model configuration.

To move from testing to a live production environment, deploy your runner to Modal.

1. Run the deployment command:

   modal deploy runner

2. Verify the deployment in your Modal dashboard.
3. Use the API_URL provided by Modal to interact with your live model via API calls.

Once the runner is active and your environment is configured, you can execute test scripts located in the scripts directory.

1. Navigate to the project root.
2. Load your environment variables:

   source .env.dev

3. Run a test script (e.g., test-simple.ts) by passing the model identifier as an argument:

   pnpm x scripts/test-simple.ts YourModel/Identifier

Tip: For easier reading during initial tests, set stream: false in your script to disable streaming and receive the full response at once.

Expected Output Format: The script returns a JSON object containing the generated text and token usage:

{
  "text": "...",
  "prompt_tokens": 23,
  "completion_tokens": 770,
  "done": true
}

The Runner requires several secret groups to function correctly. Use the modal secret create command to set these up:

To disable optional services like Sentry or Datadog, provide an empty value for the required key (e.g., SENTRY_DSN=).

The /generation endpoint allows you to generate 3D assets from a text prompt using the Shap-E model. It accepts a JSON payload and returns a list of Generation objects containing a base64-encoded Data URI of the generated .ply mesh.

Authentication: You must provide an API key in the X-API-Key HTTP header.

Request Body (Input):

• prompt (string): The text description for the 3D asset.
• num_outputs (integer): The number of variations to generate.
• num_inference_steps (integer): The number of diffusion steps for inference.
• extension (string, optional): The file extension.

Response Body (Generation):

• uri (string, optional): A base64-encoded Data URI in the format data:application/x-ply;base64,....
• url (string, optional): An optional URL to the asset.

The Shap-E application uses header-based API key authentication. To authorize your requests, include the X-API-Key header with your assigned key.

Header Name: X-API-Key Error Code: If the key is missing or incorrect, the API returns a 401 Unauthorized error.

The API uses the following Pydantic models for request and response validation:

Input Model Used in the POST /generation request body:

• prompt: str
• num_outputs: int
• num_inference_steps: int
• extension: Optional[str] = None

Generation Model Returned in the response list:

• uri: Optional[str] = None (Base64 Data URI for the .ply mesh)
• url: Optional[str] = None