Infinity

To run Infinity in offline environments or with models requiring specific Python dependencies (e.g., nomic-ai/nomic-embed-text-v1.5), you should build a custom image using docker buildx build. This allows you to pre-download models and install extra packages during the build stage.

Key build arguments:

• --build-arg MODEL_NAME: The model ID to download.
• --build-arg ENGINE: The inference engine (e.g., torch).
• --build-arg EXTRA_PACKAGES: A string of additional Python packages to install (e.g., "torch_geometric").

Target the production-with-download stage in the Dockerfile.

# clone the repo
git clone https://github.com/michaelfeil/infinity
git checkout tags/0.0.52
cd libs/infinity_emb

# build download stage using docker buildx buildkit.
docker buildx build --target=production-with-download \
--build-arg MODEL_NAME=michaelfeil/bge-small-en-v1.5 --build-arg ENGINE=torch \
-f Dockerfile -t infinity-model-small .

# Example with extra packages:
# docker buildx build --target=production-with-download \
# --build-arg MODEL_NAME=michaelfeil/bge-small-en-v1.5 --build-arg ENGINE=torch \
# --build-arg EXTRA_PACKAGES="torch_geometric" \
# -f Dockerfile -t infinity-model-small .

Use the InfinityEmbeddingsLocal class to run embeddings locally without an external API server.

Important Usage Notes:

• Lifecycle Management: You MUST use the async with statement to manage the engine. This starts and stops the batching engine correctly. Avoid frequently closing and starting the engine; instead, keep it running for the duration of your tasks.
• Manual Control: If you need more granular control, you can manually call await embeddings.__aenter__() and await embeddings.__aexit__().
• Hardware: Set device="cuda" for AMD/Nvidia GPUs via torch.

from langchain.embeddings.infinity import InfinityEmbeddingsLocal
from langchain.docstore.document import Document

embeddings = InfinityEmbeddingsLocal(
    model="sentence-transformers/all-MiniLM-L6-v2",
    # revision
    revision=None,
    # best to keep at 32
    batch_size=32,
    # for AMD/Nvidia GPUs via torch
    device="cuda",
    # warm up model before execution,
)

documents = [Document(page_content="Hello world!", metadata={"source": "unknown"})]

# important: use engine inside of `async with` statement to start/stop the batching engine.
async with embeddings:
    # avoid closing and starting the engine often.
    # rather keep it running.
    # you may call `await embeddings.__aenter__()` and `__aexit__()`
    # if you are sure when to manually start/stop execution` in a more granular way
    documents_embedded = await embeddings.aembed_documents(documents)
    query_result = await embeddings.aembed_query(query)
    print("embeddings created successful")
print(documents_embedded, query_result)

To deploy Infinity on AWS ECS, your task definition must include specific configurations for Neuron hardware access:

1. Placement Constraints: Ensure tasks are placed on instances with the correct type (e.g., inf2.xlarge).
2. Linux Parameters: You must map the device /dev/neuron0 from the host to the container and grant IPC_LOCK capabilities.
3. Port Mappings: Define the port used by the infinity_emb process.

{
    "family": "ecs-infinity-neuron",
    "requiresCompatibilities": ["EC2"],
    "placementConstraints": [
        {
            "type": "memberOf",
            "expression": "attribute:ecs.os-type == linux"
        },
        {
            "type": "memberOf",
            "expression": "attribute:ecs.instance-type == inf2.xlarge"
        }
    ],
    "executionRoleArn": "${YOUR_EXECUTION_ROLE}",
    "containerDefinitions": [
        {
            "entryPoint": ["infinity_emb", "v2"],
            "portMappings": [
                {
                    "hostPort": 7997,
                    "protocol": "tcp",
                    "containerPort": 7997
                }
            ],
            "linuxParameters": {
                "devices": [
                    {
                        "containerPath": "/dev/neuron0",
                        "hostPath": "/dev/neuron0",
                        "permissions": ["read", "write"]
                    }
                ],
                "capabilities": {
                    "add": ["IPC_LOCK"]
                }
            },
            "cpu": 0,
            "memoryReservation": 1000,
            "image": "infinity-neuron:latest",
            "essential": true,
            "name": "infinity-neuron"
        }
    ]
}