FastMLX Documentation

To contribute to FastMLX, follow these steps to set up your local development environment:

1. Fork the repository on GitHub.
2. Clone your fork locally:

   git clone git@github.com:your_name_here/fastmlx.git

3. Set up a virtual environment and install the package in development mode. Using virtualenvwrapper:

   mkvirtualenv fastmlx
   cd fastmlx/
   python setup.py develop

   Note: If you don't have virtualenvwrapper, install it via pip install virtualenvwrapper.
4. Create a feature branch:

   git checkout -b name-of-your-bugfix-or-feature

$ git clone git@github.com:your_name_here/fastmlx.git

$ mkvirtualenv fastmlx
$ cd fastmlx/
$ python setup.py develop

$ git checkout -b name-of-your-bugfix-or-feature

Before committing changes, ensure your code adheres to project standards by running flake8 for linting and pytest for testing. You should also ensure compatibility with different Python versions using tox.

First, install the necessary tools:

pip install flake8 tox

Then, run the checks:

flake8 fastmlx tests
pytest .

$ flake8 fastmlx tests
$ pytest .

The --reload flag enables automatic server reloading, which is useful during development.

Important Constraints:

• --reload only works when --workers is set to None (i.e., not explicitly configured or using the default single-process behavior).
• This option is intended for development only and should not be used in production environments.

fastmlx --reload

To install the latest stable release of FastMLX, use the standard pip installation command. This is the recommended method as it ensures you receive the most recent stable version and automatically handles required dependencies.

pip install -U fastmlx

Install FastMLX using pip to host MLX models (LMs and VLMs) with an OpenAI-compatible API.

pip install fastmlx

If you need to install FastMLX directly from the source code repository, you can use pip to install from the GitHub URL.

pip install git+https://github.com/Blaizzy/fastmlx

You can start the FastMLX server using either the built-in CLI command or by invoking uvicorn directly.

Development Mode: Use the --reload flag with uvicorn for automatic restarts during development.

Warning: Do NOT use the --reload flag in a production environment.

# Option 1: Using the fastmlx CLI
fastmlx

# Option 2: Using uvicorn directly (Development mode)
uvicorn fastmlx:app --reload --workers 0

When your changes are ready, follow these steps to submit a Pull Request:

1. Commit and push your changes:

   git add .
   git commit -m "Your detailed description of your changes."
   git push origin name-of-your-bugfix-or-feature

2. Open a Pull Request on the FastMLX GitHub repository.

Pull Request Guidelines:

• Include tests for your changes.
• If adding functionality, update the documentation. New functionality should be placed in a function with a docstring, and the feature should be added to the list in README.rst.
• Ensure the code works for Python 3.8 and later, as well as PyPy.

$ git add .
$ git commit -m "Your detailed description of your changes."
$ git push origin name-of-your-bugfix-or-feature

To improve throughput and handle multiple requests simultaneously, you can configure the number of worker processes.

Precedence (Highest to Lowest):

1. Explicit --workers command-line argument.
2. FASTMLX_NUM_WORKERS environment variable.
3. Default value of 2.

Worker Configuration Options:

• Absolute number: --workers 4 sets exactly 4 workers.
• Fraction of CPU cores: --workers 0.5 sets workers to half the available CPU cores (minimum of 1). Use 1.0 to use all available cores.

Considerations:

• The application must be stateless.
• Ensure database connection pooling is configured for multiple workers.
• Incoming requests are automatically load-balanced across workers.

# Set 4 workers via CLI
fastmlx --workers 4

# Set 4 workers via uvicorn
uvicorn fastmlx:app --workers 4

For standard Language Models (LLMs), use the /v1/chat/completions endpoint with a messages array. The API follows the OpenAI chat completion format.

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/gemma-2-9b-it-4bit",
    "messages": [{"role": "user", "content": "What is the capital of France?"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())

For Vision Language Models (VLMs), include an image field (URL or path) in your request payload to the /v1/chat/completions endpoint.

import requests
import json

url = "http://localhost:8000/v1/chat/completions"
headers = {"Content-Type": "application/json"}
data = {
    "model": "mlx-community/nanoLLaVA-1.5-4bit",
    "image": "http://images.cocodataset.org/val2017/000000039769.jpg",
    "messages": [{"role": "user", "content": "What are these"}],
    "max_tokens": 100
}

response = requests.post(url, headers=headers, data=json.dumps(data))
print(response.json())