repository·main·Indexed 25 days ago
https://github.com/ianarawjo/chainforgeAn open-source visual data flow environment for systematic prompt engineering, comparison, and evaluation of Large Language Model (LLM) responses. ChainForge allows developers to test prompt permutations and model settings simultaneously, supporting a wide range of providers including OpenAI, Anthropic, Google Gemini, and Amazon Bedrock. It includes features for automated grading via the evalgen module, integration with OpenAI evals, and tools for generating and autofilling data tables using LLMs.
The evalgen module automates the process of finding the best evaluation functions for a given prompt and set of examples. It works by:
You can set your API keys (OpenAI, Anthropic, Google, AWS, etc.) by clicking the Settings icon in the top-right corner of the ChainForge interface.
Recommendation: To avoid re-entering keys every time you open ChainForge, save your API keys and/or Amazon AWS credentials to your local environment variables. The locally installed version can load these automatically.
npm start command. This will run the app in development mode and make it available at http://localhost:3000. The page will automatically reload when you make changes, and lint errors will be displayed in the console.npm start.xlsx format), attach an Inspect node to the output of a Prompt node and click the Export Data button.You can run ChainForge locally using Docker Desktop by building the provided Dockerfile and running the resulting image. Map port 8000 to access the interface at http://127.0.0.1:8000.
docker build -t chainforge .
docker run -p 8000:8000 chainforgeThe web version at https://chainforge.ai/play/ allows you to share your experiment flows using a unique link. Click the Share button to generate and copy a link to your clipboard.
Limitations:
.cforge files.If you need full control over the build tool and configuration (webpack, Babel, ESLint, etc.), you can use npm run eject.
Warning: This is a one-way operation. Once you eject, you cannot go back.
Executing this command removes the single build dependency and copies all configuration files and transitive dependencies directly into your project.
npm run ejectevalgen functionality, run the provided test.ts script using ts-node. Note that the terminal output may experience occasional lag.npm run build command. This command bundles React in production mode and optimizes the build for performance, outputting the result to the build folder. The build is minified and includes file hashes in the filenames for deployment readiness.npm run buildnpm test command.npm testTo install ChainForge on your machine, ensure you have Python 3.8 or higher installed. You can then install the package using pip and start the server using the chainforge serve command. Once running, access the interface at http://localhost:8000 in a supported browser (Chrome, Firefox, Edge, or Brave).
pip install chainforge
chainforge serveChainForge provides access to a subset of OpenAI's evals registry (derived from https://github.com/openai/evals). These evals are provided as ChainForge flows.
To keep the core chainforge PyPI package lightweight, these files are not bundled with the installation. Instead, they are fetched from GitHub on an as-needed basis when you attempt to use them. This avoids the requirement for users to manually install the full OpenAI evals suite, which requires Git LFS, Python 3.9+, and numerous heavy dependencies.