unbody

Before setting up the Unbody development environment, ensure you have the following installed on your system:

• Nix package manager
• Docker with the Docker Compose plugin
• Git

To start developing, you need to run both the required infrastructure services and the application server.

1. Start infrastructure services (e.g., databases, caches) using Docker Compose:

docker compose up

2. Start the development server using Yarn:

yarn start:dev

To install the Unbody repository, clone it from GitHub and use yarn to install dependencies. Note that npm is not recommended as it may not install dependencies correctly.

git clone https://github.com/unbody-io/unbody
cd unbody
yarn

Set up your local environment by copying the example environment file to .env.local and configuring your required keys (such as your OpenAI API key).

cp .env.example .env.local
# Edit .env.local with your credentials

Follow these steps to clone the repository and initialize your local development environment using Nix and Yarn.

1. Clone the repository

git clone https://github.com/unbody-io/opensource.git
cd unbody

2. Enable Nix flakes

If you are using the Determinate Nix Installer, flakes are enabled by default. Otherwise, follow the NixOS Wiki Flakes guide.

3. Enter the development shell

Run the following command to enter a shell pre-configured with Node.js 22 (LTS), Yarn, and local node_modules/.bin in your PATH:

nix develop

4. Install dependencies

Once inside the Nix shell, install the project dependencies:

yarn install

To run the application, start the required services using Docker Compose in detached mode, then start the Unbody application using yarn start.

docker compose up -d
yarn start

When implementing a provider, you can define the structure of your source data using ProviderContextSourceData. This type encapsulates the identity and state of the data source within the plugin context.

It consists of:

• id: A unique identifier for the source.
• state: The current internal state of the source (e.g., last sync timestamp, cursor).
• entrypoint: The specific configuration or subset of data being targeted.
• credentials: The authentication data required to access the source.

export type ProviderContextSourceData<\n  E extends Record<string, any> = Record<string, any>, // Entrypoint\n  C extends Record<string, any> = Record<string, any>, // Credentials\n  S extends Record<string, any> = Record<string, any>, // State\n> = {\n  id: string\n  state: S\n  entrypoint: E\n  credentials: C\n}

The img2vec-neural service uses the semitechnologies/img2vec-pytorch:resnet50 image. It can be configured via environment variables, such as disabling CUDA support.

img2vec-neural:
  image: semitechnologies/img2vec-pytorch:resnet50
  environment:
    ENABLE_CUDA: '0'

The unbody.config.json file allows you to define a list of plugins to be used by the Unbody engine. Each plugin is defined as an object within the plugins array. While the current schema shown is minimal, this is where you would specify plugin configurations to extend Unbody's capabilities.

{
  "plugins": [{}]
}

When using the Weaviate service via Docker Compose, you can configure its behavior using several environment variables. Key settings include module enablement, authentication, and persistence paths.

Note that ENABLE_MODULES is used to activate specific Unbody-related modules like generative-unbody, reranker-custom, and multi2vec-custom.

weaviate:
  environment:
    - QUERY_DEFAULTS_LIMIT=25
    - AUTHENTICATION_ANONYMOUS_ACCESS_ENABLED=true
    - PERSISTENCE_DATA_PATH=/var/lib/weaviate
    - CLUSTER_HOSTNAME=node1
    - ENABLE_MODULES=text2vec-huggingface,generative-unbody,img2vec-custom,reranker-custom,multi2vec-custom
    - CLUSTER_GOSSIP_BIND_PORT=7100

Use the unbody-cli to add new data sources to your indexing pipeline.

yarn unbody-cli source add