Marqo AI-Native Ecommerce Search Platform

Multimodal search allows you to operate over multiple data types simultaneously, such as text and images. In Marqo, this is achieved through two primary methods:

1. Multimodal Queries: Using a combination of text and images to perform a search.
2. Multimodal Documents: Indexing documents that contain both text and images.

By combining these modalities, you can capture complementary information that a single modality might miss (e.g., using text to disambiguate the subject of an image). This approach enables advanced features like searching via natural language prompting, per-query personalization, and incorporating business logic or relevance feedback directly into the search experience without retraining models.

You can steer search results toward specific themes or items (e.g., items a user has liked or purchased) by using a context vector. This acts as a form of relevance feedback. To avoid inference latency at search time, you can pre-compute the vectors for a set of items and fuse them into a single context vector to be used alongside your primary query.

# Conceptual representation of combining a query with a pre-computed context vector
query = {"backpack": 1.0}
context_vector = [.1, ..., -.8] # Pre-computed from a set of items

When augmenting an LLM with retrieved context, you can use Marqo's search highlights to perform token-aware truncation. This ensures that the text provided to the LLM includes relevant context from immediately before and after the most relevant match, helping to stay within token limits while maintaining semantic coherence.

In the provided workflow, the extract_text_from_highlights function (a utility used in the example) takes the search results and a token_limit to return the specific text segments to be used as summaries in a Langchain prompt.

# Example of preparing context from search results
highlights, texts = extract_text_from_highlights(results, token_limit=150)

# Formatting for Langchain Document objects
docs = [Document(page_content=f"Source [{ind}]:" + t) for ind, t in enumerate(texts)]

You can perform zero-shot classification by creating a separate index containing only your category labels (e.g., [{"label": "a face"}, {"label": "a hamburger"}]). By searching for an image against this label index, the returned _score for each label represents the confidence of that classification.

# Create a label index
labels = [{"label": "one hot dog"}, {"label": "two hot dogs"}, {"label": "a hamburger"}, {"label": "a face"}]
client.create_index(index_name, settings_dict=settings)
client.index(index_name).add_documents(labels, tensor_fields=["label"])

# Classify an image by searching for it in the label index
# The returned hits will contain the scores for each label
responses = client.index(index_name).search(image_url, device='cpu')

for hit in responses['hits']:
    print(f"Label: {hit['label']}, Score: {hit['_score']}")

Depending on your environment, you can run the service in development mode or production mode.

Development mode: Run the module directly using python -m.

Production mode: Use uvicorn to serve the FastAPI application on a specific host and port.

Docker: You can also run the service as a container.

Requirements:

• NVIDIA Triton inference server (required for production deployments)

# Development mode
PYTHONPATH=./src python -m inference_orchestrator.main

# Production mode
PYTHONPATH=./src uvicorn inference_orchestrator.main:app --host 0.0.0.0 --port 8884

# Docker build and run
docker build -t marqo-inference .
docker run -p 8884:8884 marqo-inference

Documents must be formatted as Python dictionaries for ingestion. Each dictionary should contain a text field and optional metadata like source. After preparing your list of documents, use the marqo.Client to create an index and ingest the data. If no specific encoder is provided, Marqo uses its default encoder.

from marqo import Client

# Prepare documents
document1 = {"text":"Auto-Off function: This feature automatically switches off the steam iron if it has not been moved for a while.", "source":"page 1"}
documents = [document1, document2, document3, document4, document5]

# Create index and ingest
mq = Client()
index_name = "iron-docs"
mq.create_index(index_name)
# Note: In a full implementation, you would call mq.index(index_name).add_documents(documents)

To simulate a distributed environment, you can run a multi-node Vespa cluster. The cluster composition is determined by the number of shards and replicas provided:

• Config nodes: 3 nodes.
• Content nodes: m nodes, where m = number_of_shards * (1 + number_of_replicas).
• API nodes: n nodes, where n = max(2, number_of_content_nodes).

Example: Using --Shards 2 --Replicas 1 results in 4 content nodes and 2 API nodes.

python vespa_local.py start --Shards 2 --Replicas 1