MemoryAgentBench Documentation

To set up a reproducible environment for MemoryAgentBench, follow these steps:

1. Create a Conda Environment:

   conda create --name MABench python=3.10.16

2. Install Dependencies: Install PyTorch, the project requirements, and a specific version of NumPy to avoid compatibility issues:

   pip install torch
   pip install -r requirements.txt
   pip install "numpy<2"

Note on Package Conflicts:

• hipporag is not included in requirements.txt due to version conflicts. If you need it, create a separate environment.
• If you encounter errors related to cognee or letta, try reinstalling them as follows:

  pip install letta
  pip uninstall letta   
  pip install cognee
  pip uninstall cognee

Datasets can be automatically downloaded when running the code, or you can manually download them from the HuggingFace repository: ai-hyz/MemoryAgentBench.

Important: Ensure you include entity2id.json for the Movie Recommendation task.

You can run different types of evaluations using the provided shell scripts in bash_files/eniac/.

Long Context Agents

bash bash_files/eniac/run_memagent_longcontext.sh

Arguments:

• --agent_config: Path to the agent/model configuration file.
• --dataset_config: Path to the dataset configuration file.

RAG Agents and Agentic Memory Methods

bash bash_files/eniac/run_memagent_rag_agents.sh

Ablation Study for Chunk Size

bash bash_files/eniac/run_memagent_rag_agents_chunksize.sh

Warning: hipporag (2.0.0a3) requires openai==1.58.1, which may prevent the use of the latest OpenAI models in the same environment.

For specific metric evaluations using an LLM-as-a-judge, run the following Python scripts (ensure you have configured the necessary settings in your environment/configs):

• LongmemEval: python llm_based_eval/longmem_qa_evaluate.py
• InfBench Summarization: python llm_based_eval/summarization_evaluate.py

The run_agent_loop function manages the lifecycle of an interactive session between a user and a letta.agent.Agent.

1. Initialization: It creates a create_client() instance and determines if the user or the agent starts the conversation.
2. Input Loop: It uses questionary to capture user text. If // is entered, it toggles multiline mode.
3. Command Processing: If input starts with /, it executes internal commands (like /save, /memory, or /attach) instead of sending the text to the LLM.
4. Agent Execution: If the input is a standard message, it calls letta_agent.step_user_message().
5. System Feedback: The loop automatically handles system-level responses such as token_warning (sending a memory warning) or function_failed (sending a heartbeat).

Create a .env file in the project root to configure your LLM provider API keys. The following keys are required for different components:

• OpenAI: OPENAI_API_KEY
• Cognee: LLM_MODEL and LLM_API_KEY
• Anthropic: Anthropic_API_KEY
• Google: Google_API_KEY

OPENAI_API_KEY=###your_openai_api_key

# Settings for Cognee
LLM_MODEL=gpt-4o-mini
LLM_API_KEY=###your_api_key

# Other API Keys
Anthropic_API_KEY=###your_anthropic_api
Google_API_KEY=###your_google_api

The mem0.proxy module provides a high-level abstraction over the standard mem0 library and litellm:

• Mem0: The entry point. It manages the lifecycle of either a local Memory instance or a remote MemoryClient.
• Chat: A container for chat-related operations.
• Completions: The core logic engine. It handles the orchestration between:
  • Message Preparation: Ensures a system prompt containing MEMORY_ANSWER_PROMPT is present.
  • Asynchronous Memory Updates: Uses a background thread to call mem0_client.add() so that memory storage doesn't block the LLM response.
  • Context Injection: Fetches relevant facts from memory and reformats the user's query to include Relevant Memories/Facts and Entities before sending it to the LLM.

When the server is running in secure mode (triggered by LETTA_SERVER_SECURE=true or the --secure flag), it uses CheckPasswordMiddleware to protect endpoints.

To bypass the 401 Unauthorized error, you must provide the server's randomly generated password in one of two ways via HTTP headers:

1. X-BARE-PASSWORD: X-BARE-PASSWORD: password <YOUR_PASSWORD>
2. Authorization: Authorization: Bearer <YOUR_PASSWORD>

Note: The /v1/health/ and /latest/health/ endpoints are excluded from this protection.

The API uses a custom CogneeApiError for structured error responses. When an error occurs, the API returns a JSON response with a detail field containing the error message.

If the error is improperly defined (missing name, message, or status_code), the server defaults to a 418 I'm a teapot status code with the message An unexpected error occurred.