DOCSEARCHbyAlgolia
Submit Docs

Modular Cloud

website·Indexed 37 days ago

https://www.modular.com/blog

A powerful inference solution for deploying and running AI models. Modular Cloud provides APIs for text, image, and video generation, including support for function calling and reasoning. The platform offers dedicated deployments for organizations, management of API keys, roles, and billing, and integrations with coding assistants such as Cursor, TRAE, OpenCode, and Hermes Agent.

Tokens
24.8K
Snippets
96
Records
174
Agent score
71%

What's inside Modular Cloud

  1. Overview of Modular Cloud AI Infrastructure

    Modular Cloud provides a unified AI inference stack designed to optimize the cost-to-performance ratio for AI use cases. Unlike fragmented infrastructure, Modular integrates the entire stack from kernels to the cloud, enabling flexible deployment across NVIDIA, AMD, and other hardware architectures.

  2. Overview of Modular Cloud Inference

    Modular Cloud provides a unified inference solution designed to optimize the cost-to-performance ratio for AI use cases. Unlike fragmented AI infrastructure, Modular uses a unified stack from kernels to the cloud, enabling flexible deployment across various hardware including NVIDIA and AMD GPUs.

  3. Understand Modular Cloud endpoint types

    Modular Cloud offers two types of endpoints for model inference:

    • Shared endpoints: Serverless, pay-as-you-go inference on multi-tenant infrastructure. These are automatically available to all organizations.
    • Dedicated endpoints: Provisioned via dedicated deployments. These are recommended for production inference and can be created for any supported model.

  4. Generate images or videos using the Modular Cloud Inference API

    Use the POST /v1/responses endpoint to generate visual content. This endpoint accepts text prompts or images as input to produce a new image or video. Before using this endpoint, verify which models support these specific modalities by checking the Supported Models documentation.

  5. Understand and request Dedicated Deployments on Modular Cloud

    Dedicated deployments provide infrastructure isolated to a single organization. Each deployment runs one specific model and exposes a private endpoint accessible only to that organization. This is distinct from shared endpoints, which are available to all users via an API key.

    To set up a dedicated deployment, you must contact Modular via their request demo page.

  6. Understand and request Dedicated Deployments

    Dedicated deployments provide infrastructure isolated to a specific organization. Each deployment runs a single model and exposes a private endpoint accessible only to that organization. This is distinct from shared endpoints, which are available to all users via an API key.

    Dedicated deployments are available exclusively on the Enterprise plan.

  7. Understand how Modular Cloud rate limits work

    Inference requests are governed by two levels of rate limits. A request only succeeds if there is available capacity in both:

    1. Tier limits: Organization-wide budgets that apply to all requests across every model.
    2. Model limits: Independent limits enforced per endpoint for specific models.

    If either limit is exceeded, the API returns an HTTP 429 response. To increase limits, users should contact Modular via the console.

  8. Understand Modular Cloud endpoint types

    Modular Cloud offers two types of endpoints for model inference:

    • Shared endpoints: Serverless, pay-as-you-go inference on multi-tenant infrastructure. These are automatically available to all organizations.
    • Dedicated endpoints: Provisioned via dedicated deployments. These are recommended for production inference and can be created for any supported model.

  9. Understand how Modular Cloud rate limits work

    Rate limits determine the maximum number of inference requests an organization can send, measured in Requests Per Minute (RPM) and Tokens Per Minute (TPM). Limits are enforced at two levels; a request only succeeds if both have available capacity. If either limit is exceeded, the API returns an HTTP 429 response.

    • Tier limits: Organization-wide budgets that apply to all requests across every model.
    • Model limits: Independent limits enforced per endpoint for specific models.

  10. Configure Modular Cloud in Hermes Desktop

    To add Modular Cloud as a provider in the Hermes Desktop application:

    1. Open the app and click the gear icon (Settings).
    2. Navigate to Providers > Custom Endpoints in the left sidebar.
    3. In the Add Endpoint form, enter:
      • Name: Modular
      • Provider ID: modular
      • Endpoint URL: https://api.modular.com/v1
      • Default Model: A model you have access to (e.g., minimax/minimax-m3)
      • API Key: Your Modular Cloud API key
    4. Click Test to verify the connection, then click Save.

  11. Connect Cursor to Modular Cloud models

    To use Modular Cloud models in Cursor, you must configure the OpenAI API settings to point to the Modular Cloud endpoint. Note that custom models are supported in Chat, Composer, and Edit modes, but Tab autocomplete continues to use Cursor's built-in model and does not support custom endpoints.

    Prerequisites:

    • A Modular Cloud account
    • A Modular Cloud API key
    • Cursor installed

  12. Authenticate OpenCode with Modular Cloud

    To connect OpenCode to Modular Cloud, you must first create a Modular Cloud API key. Once you have the key, start OpenCode by running the opencode command, then execute the /connect command within the application. When prompted for a provider, select 'Other', set the Provider id to modular, and enter your API key.
    opencode