DOCSEARCHbyAlgolia
Submit Docs

OpenAI Forward

repository·main·Indexed 21 days ago

https://github.com/kenyony/openai-forward

A high-performance asynchronous reverse proxy service for Large Language Models (LLMs). It provides advanced features including request and token-based rate limiting, smart caching, API key management, and multi-target routing for OpenAI-compatible, local (e.g., LocalAI), and general cloud models (e.g., Gemini). The service includes a CLI tool `aifd` and an optional WebUI for configuration management.

Tokens
7.1K
Snippets
39
Records
41
Agent score
76%

What's inside openai-forward

  1. Enable intelligent prediction caching

    main

    OpenAI-Forward supports caching AI predictions to accelerate access and reduce costs.

    When using openai forwarding mode, you can control caching behavior per request using the extra_body parameter in the Python SDK or by adding a caching key to the JSON body in a raw request.

    Python SDK Example:

    completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"caching": True}
)

    Curl Example:

    curl https://smart.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-******" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}],
    "caching": true
  }'
    client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"caching": True}
)

  2. Use Smart Caching for AI predictions

    main

    OpenAI Forward supports caching AI responses to speed up access and save costs.

    • General Forwarding (type="general"): When caching is enabled, the service automatically returns cached responses for identical requests.
    • OpenAI Forwarding (type="openai"): You control caching behavior per request using the extra_body parameter (Python) or by adding a caching key to the JSON body (Curl).

    Python Example:

    from openai import OpenAI 
client = OpenAI(
    base_url="https://smart.openai-forward.com/v1", 
    api_key="sk-******"
)
completion = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_body={"caching": True}
)

    Curl Example:

    curl https://smart.openai.com/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer sk-******" \
  -d '{
    "model": "gpt-3.5-turbo",
    "messages": [{"role": "user", "content": "Hello!"}],
    "caching": true
  }'
    extra_body={"caching": True}

  3. Use Custom Secret Keys

    main

    OpenAI-Forward allows you to use custom secret keys (e.g., starting with fk-) instead of the original OpenAI API keys. This is useful for managing access through the proxy service.

    import openai
openai.api_base = "https://api.openai-forward.com/v1"
openai.api_key = "fk-******"

  4. Deploy using Docker

    main

    Run the service as a container using the official image. This maps the host's port 8000 to the container's port 8000.

    • Image: beidongjiedeguang/openai-forward:latest
    • Access URL: http://{ip}:8000
    • Internal Log Path: /home/openai-forward/Log/ (this path can be mapped to a host volume during startup).
    docker run -d -p 8000:8000 beidongjiedeguang/openai-forward:latest

  5. Test proxy forwarding performance

    main

    To evaluate the performance of openai-forward when proxying requests (both streaming and non-streaming), you must point the proxy to a backend server (e.g., the FastAPI benchmark server).

    Setup

    Set OPENAI_BASE_URL to the address of your backend server and start the proxy:

    OPENAI_BASE_URL=http://localhost:8080 aifd run --workers=n --port 8000

    (Replace n with the number of workers, e.g., 1 or 4).

    Running Benchmarks with wrk

    Test the proxy endpoint at http://localhost:8000/benchmark/v1/chat/completions using the following command:

    wrk -t8 -c100 -d10s -s post.lua http://localhost:8000/benchmark/v1/chat/completions

    This benchmark allows you to compare performance across different HTTP clients used for forwarding, such as httpx and aiohttp, for both streaming and non-streaming modes.

    OPENAI_BASE_URL=http://localhost:8080 aifd run --workers=n --port 8000

  7. Test FastAPI native streaming and non-streaming performance

    main

    To evaluate the performance of the underlying fastapi framework itself (without proxying/forwarding), use the BENCHMARK_MODE=true environment variable. This tests how the server handles direct requests for streaming and non-streaming completions.

    Setup

    Start the server with the following command:

    BENCHMARK_MODE=true aifd run --workers=n --port 8080

    (Replace n with the number of workers, e.g., 1 or 16).

    Running Benchmarks with wrk

    Use the wrk tool with a post.lua script to test the /benchmark/v1/chat/completions endpoint.

    Non-streaming (stream == false)

    Single core:

    wrk -t8 -c400 -d10s -s post.lua http://localhost:8080/benchmark/v1/chat/completions

    16 cores:

    wrk -t15 -c500 -d10s -s post.lua http://localhost:8080/benchmark/v1/chat/completions

    Streaming (stream == true)

    Single core:

    wrk -t8 -c100 -d10s -s post.lua http://localhost:8080/benchmark/v1/chat/completions

    Note: To simulate real-world scenarios, you can configure different TOKEN_RATE_LIMIT values. Streaming text data is located in cache/chat.

    BENCHMARK_MODE=true aifd run --workers=n --port 8080

  8. Deploy via pip

    main

    Install the openai-forward package using pip and run the service using the aifd CLI tool. You can choose to run the standard service or the version with a WebUI.

    # Install
pip install openai-forward

# Run standard service
aifd run

# Run with WebUI
aifd run --webui

  9. One-click deployment to Render

    main

    Render is a recommended method for free cloud deployment.

    Deployment Steps

    1. Click the Deploy to Render button or fork the repository and connect it to a new Render Web Service.
    2. Configure environment variables (all openai-forward settings can be passed via environment variables).
    3. Wait for the build to complete.

    Important Render Considerations

    • Sleep Mode: On the free plan, Render services sleep after 15 minutes of inactivity. The first request after sleep will experience a ~15s delay. To prevent sleeping, use a keep-alive script (see scripts/keep_render_alive.py in the repo).
    • Zero-Downtime: Set the Health Check Path in Render settings to /healthz to ensure smooth deployments.

  10. Configure Chat Logs and convert to JSON

    main

    Chat logs are disabled by default. To enable them, set the LOG_CHAT=true environment variable. Logs are stored in the current directory under Log/openai/chat/chat.log.

    To convert the text-based logs into a structured JSON format, use the aifd convert command. This will generate a chat_openai.json file.

    aifd convert

  11. Enable SSL for HTTPS access

    main

    To access your service via a domain name using HTTPS, use a reverse proxy like Nginx (or Nginx Proxy Manager) or Caddy to forward traffic from the HTTPS port to the openai-forward service port (default 8000).

    Important for Streaming: If you are using streaming responses, you must disable proxy buffering in your Nginx configuration to prevent issues. In Nginx Proxy Manager, add this to the 'Custom Nginx Configuration' field:

    proxy_buffering off;

  12. Proxy arbitrary cloud models (e.g., Gemini)

    main

    You can proxy any cloud service by setting the FORWARD_CONFIG environment variable. For services that do not follow the OpenAI format, use type="general".

    Example for Gemini Pro:

    FORWARD_CONFIG=[{"base_url":"https://generativelanguage.googleapis.com","route":"/gemini","type":"general"}]

    After starting the service, you can access Gemini via http://localhost:8000/gemini.

    FORWARD_CONFIG=[{"base_url":"https://generativelanguage.googleapis.com","route":"/gemini","type":"general"}]