DOCSEARCHbyAlgolia
Submit Docs

LoCoMo Benchmark

repository·main·Indexed 21 days ago

https://github.com/snap-research/locomo

A high-quality evaluation benchmark for assessing the very long-term conversational memory of LLM agents. LoCoMo provides datasets for question-answering, event summarization, and multimodal dialog generation, including tools to generate long-term conversations between agents with custom or MSC dataset personas and scripts to evaluate model providers including OpenAI, Anthropic, Gemini, and Huggingface.

Tokens
827
Snippets
2
Records
6
Agent score
27%

What's inside LoCoMo

  1. Understand the LoCoMo dataset structure

    main

    The LoCoMo dataset is a benchmark of very long-term conversational data stored in ./data/locomo10.json. Each sample in the JSON file represents a single conversation and includes the following fields:

    • sample_id: Unique identifier for the sample.
    • conversation: A list of sessions (session_<num>) and their timestamps (session_<num>_date_time). It includes speaker names (speaker_a, speaker_b) and turns. Each turn contains the speaker, dialog ID (dia_id), and text. If images are present, it includes img_url, blip_caption, and the icrawler search query.
    • observation (generated): Session-level observations (session_<num>_observation) used for RAG evaluation.
    • session_summary (generated): Session-level summaries (session_<num>_summary) used for RAG evaluation.
    • event_summary (annotated): Ground truth significant events for each speaker within a session (events_session_<num>).
    • qa (annotated): Question-answering annotations containing question, answer, category, and evidence (list of dialog IDs).

  2. Regenerate observations and session summaries

    main

    If you need to regenerate the generated fields for RAG evaluation using gpt-3.5-turbo, use the following scripts:

    • Generate observations: bash scripts/generate_observations.sh (generates session_<num>_observation for all sessions).
    • Generate session summaries: bash scripts/generate_session_summaries.sh (generates session_<num>_summary).

    Note: Session summaries are single-session summaries, whereas event_summary (annotated) contains causal and temporal connections across sessions.

  3. Configure environment variables for LoCoMo scripts

    main
    Before running any scripts in the repository, you must configure your environment variables (such as API keys and output directories) in scripts/env.sh. This file is sourced at the beginning of all other execution scripts to ensure the environment is correctly set up.

  4. Generate long-term conversations between LLM agents

    main

    You can generate very long-term conversations between two LLM agents with pre-assigned personalities using the generative framework in scripts/generate_conversations.sh.

    Custom Personas

    To use custom personas, point the --out-dir flag to a directory containing agent_a.json and agent_b.json. Each file must follow this format:

    {
  "name": "Angela",
  "persona_summary": "Angela is a 31 year old woman..."
}

    MSC Dataset Personas

    To sample personalities from the MSC dataset, point --out-dir to an empty directory. The script will then sample pairs from data/msc_personas_all.json.

    Parameters

    Detailed parameters (like --num-days to specify the temporal span) can be adjusted in scripts/generate_conversations.py.

    bash scripts/generate_conversations.sh

  6. Evaluate LLMs on the LoCoMo Question Answering task

    main

    Use the following scripts to evaluate different model providers on the LoCoMo QA task using the (truncated) conversation as context:

    • OpenAI models: bash scripts/evaluate_gpts.sh
    • Anthropic models: bash scripts/evaluate_claude.sh
    • Gemini models: bash scripts/evaluate_gemini.sh
    • Huggingface models: bash scripts/evaluate_hf_llm.sh