ModelFusion

ModelFusion is a vendor-neutral abstraction layer for integrating AI models into JavaScript and TypeScript applications. It unifies APIs for common AI operations and provides production-ready features.

Core Capabilities

• Multi-modal support: Handles text generation, image generation, vision, text-to-speech, speech-to-text, and embedding models.
• Unified API: Provides consistent interfaces for text streaming, object generation, and tool usage.
• Type Safety: Uses TypeScript type inference and validates model responses.
• Production Features: Includes observability hooks, logging, automatic retries, throttling, and error handling.
• Optimized for Deployment: Fully tree-shakeable and compatible with serverless environments with minimal dependencies.

Note: ModelFusion has joined Vercel and is being integrated into the Vercel AI SDK. For the latest developments in text generation, structured object generation, and tool calls, consider checking out the Vercel AI SDK.

ModelFusion provides several reference implementations and demo applications to showcase its capabilities across different runtimes and use cases. These demos cover web applications, terminal tools, agents, and edge computing.

Web & Framework Integrations

• Chatbot (Next.JS): A web chat with an AI assistant featuring streaming and abort handling using OpenAI GPT-3.5-turbo.
• Next.js / ModelFusion Demos: Demonstrates ModelFusion with Next.js 14 (App Router) for image generation, voice recording & transcription, and object streaming (using OpenAI, Stability AI, and Ollama).
• Duplex Speech Streaming: A full-stack example using Vite/React and ModelFusion Server/Fastify. It returns both text and speech streams (OpenAI and Elevenlabs) from a single prompt.
• Cloudflare Workers: Demonstrates generating text on a Cloudflare Worker using ModelFusion and OpenAI.

Agents & Terminal Apps

• BabyAGI Agent: A TypeScript implementation of the BabyAGI and BabyBeeAGI agent patterns.
• Wikipedia Agent: A ReAct agent using GPT-4 and OpenAI functions/tools to answer questions by searching Wikipedia.
• Middle school math agent: A small agent that uses a calculator tool to solve math problems via GPT-4.
• Chat with PDF: A terminal app using PDF parsing, in-memory vector indices, RAG (Retrieval Augmented Generation), and hypothetical document embedding.
• PDF to Tweet: A terminal app that parses a PDF, performs recursive information extraction, uses an in-memory vector index for style example retrieval, and calculates costs using OpenAI GPT-4.

Models provide a unified interface for different AI providers. They offer standardized APIs, configurable settings, capability information (like token limits), and fault tolerance (retries/throttling).

Creating Models

Models are created using provider facades (e.g., openai). Each provider contains factory functions like CompletionTextGenerator or ChatTextGenerator which accept a configuration object.

Configuring API settings

You can pass an api configuration object to the factory function to manage API keys, base URLs, retry strategies, and throttling.

Modifying settings with withSettings

The withSettings method creates a new model instance that inherits the original model's configuration but applies new settings (e.g., changing maxGenerationTokens).

import { openai } from "modelfusion";

// Basic model creation
const model = openai.CompletionTextGenerator({
  model: "gpt-3.5-turbo-instruct",
  maxGenerationTokens: 500,
});

// Advanced configuration with API settings
import { api, openai } from "modelfusion";

const modelWithApi = openai.CompletionTextGenerator({
  model: "gpt-3.5-turbo-instruct",
  api: openai.Api({
    apiKey: "my-api-key",
    baseUrl: {
      host: "my-proxy-host",
    },
    retry: api.retryWithExponentialBackoff({ maxTries: 5 }),
    throttle: api.throttleOff(),
  }),
});

// Creating a new model instance with updated settings
const modelWithMoreTokens = model.withSettings({
  maxGenerationTokens: 1000,
});

When rendering streamed objects, your UI components must be resilient to partial data. Because the LLM generates the object incrementally, many properties and nested values will be undefined until the stream completes.

Best Practices:

• Use optional chaining (?.) for all nested properties.
• Use nullish coalescence (??) to provide fallback values for strings.
• Use truthy checks (&&) to ensure you only render elements when their required data exists.

Example component pattern:

export const ItineraryView = ({ itinerary }: { itinerary?: Itinerary }) => (
  <div className="mt-8">
    {itinerary?.days && (
      <div className="space-y-4">
        {itinerary.days.map((day, index) => (
          day && (
            <div key={index}>
              <h3>{day.theme ?? ""}</h3>
              {day.activities?.map((activity, idx) => (
                activity && (
                  <div key={idx}>
                    <h4>{activity.name}</h4>
                    <p>{activity.description}</p>
                  </div>
                )
              ))}
            </div>
          )
        ))}
      </div>
    )}
  </div>
);

Once text is extracted from a PDF, you must index it to enable semantic retrieval. This process involves four main steps:

1. Initialize an Embedding Model: Use a model like OpenAI's text-embedding-ada-002 to convert text into vectors.
2. Chunk the Text: Use splitTextChunks with a tokenizer (from your embedding model) to break large text into smaller, manageable pieces (e.g., 256 tokens). This ensures chunks are small enough for the model but large enough to retain context.
3. Create a Vector Index: Use MemoryVectorIndex for an in-memory implementation that uses cosine similarity for searches. This is ideal for small datasets like a single PDF.
4. Upsert Chunks: Use upsertIntoVectorIndex to convert chunks into vectors and store them in the index along with their metadata (like page numbers).

This setup allows the chatbot to perform fast, semantic-based searches to find relevant context for user queries.

const pages = await loadPdfPages(file);

const embeddingModel = openai.TextEmbedder({
  model: "text-embedding-ada-002",
  throttle: throttleMaxConcurrency({ maxConcurrentCalls: 5 }),
});

const chunks = await splitTextChunks(
  splitAtToken({
    maxTokensPerChunk: 256,
    tokenizer: embeddingModel.tokenizer,
  }),
  pages
);

const vectorIndex = new MemoryVectorIndex<{
  pageNumber: number;
  text: string;
}>();

await upsertIntoVectorIndex({
  vectorIndex,
  embeddingModel,
  objects: chunks,
  getValueToEmbed: (chunk) => chunk.text,
});