ReMe (Remember Me, Refine Me)

repository·main·Indexed 25 days ago

https://github.com/agentscope-ai/reme

A local-first memory layer for AI agents that converts conversations and resources into searchable, editable Markdown files. ReMe uses a progressive system to evolve raw data into long-term, linked knowledge. It includes support for benchmarks like LongMemEval and BEAM, as well as specialized workflows such as Auto Fin for ETF event-research and Daily Paper for arXiv paper digests.

Tokens
44.3K
Snippets
105
Records
235
Agent score
84%

What's inside ReMe

  1. Overview of the ReMe Framework architecture

    main

    ReMe is a configuration-driven runtime that assembles components and Jobs to manage long-term memory. The architecture follows a hierarchical structure:

    • CLI: Parses commands and launches the service or calls it via a client.
    • Service: Exposes Jobs as HTTP endpoints or MCP tools.
    • Application: Assembles configured objects and manages their lifecycle.
    • Job: Orchestrates the execution of Steps (Normal, Streaming, Background, or Cron).
    • Step: Atomic business operations (e.g., file I/O, retrieval, indexing).
    • Component: Reusable infrastructure (e.g., file_store, keyword_index, llm).
    • Workspace: The persistent storage layer containing daily/, digest/, resource/, metadata/, and session/ directories.
  2. Overview of ReMe Automatic Memory Flow capabilities

    main

    ReMe automates the lifecycle of memory through several key capabilities:

    CapabilityEntry pointDescriptionOutput
    auto_memoryAgent hook or reme auto_memoryDistills conversation facts while preserving raw sessions.session/dialog/*.jsonl, daily/<date>/<session>.md
    auto_resourceResource watcher or reme auto_resourceTurns files in resource/<date>/ into source-linked daily cards.daily/<date>/<resource-card>.md
    auto_indexBackground watcher or reme reindexMaintains BM25, wikilink graph, and optional embedding indexes.Searchable daily/, digest/, and resource/ content
    auto_dreamdream_cron or reme auto_dreamConsolidates daily cards into long-term personal, procedure, and wiki memory.digest/**, daily/<date>/interests.yaml
    proactivereme proactiveReads topics from auto_dream to help agents decide what to mention.Structured topics from daily/<date>/interests.yaml
  3. Compare Agentic vs Prompted answer strategies

    main

    When evaluating memory performance, two primary answering strategies are used:

    • Agentic answer framework: The agent performs searches dynamically, with a limit of up to 5 search calls per question. This approach is generally more robust for complex reasoning.
    • Prompted-based answer: A fixed retrieval strategy where the original query is used to retrieve exactly 10 fileChunk items. This is faster but may have lower accuracy in multi-session or temporal reasoning tasks.
  4. Understand the ReMe workspace model

    main

    ReMe organizes memory into three distinct directory structures:

    • daily/: Lightly-processed memory containing daily facts and conversation summaries.
    • digest/: Long-term consolidated knowledge (the primary source for most recall tasks).
    • resource/: External raw materials.

    Consolidation from daily/ to digest/ and proactive interest extraction are handled automatically by the ReMe server via background watchers and a 'dream cron' process.

  5. Understand the Memory as File architecture

    main

    ReMe uses a "Memory as File, File as Memory" architecture. Long-term memory is stored as human-readable Markdown files, resource files, and index snapshots within a workspace directory rather than in a black-box database.

    Key Properties:

    • Readable & Editable: Users can manage memory using standard file operations (read, write, move, delete) without specialized database clients.
    • Traceable: Long-term conclusions can point back to original sources using derived_from:: [[...]] syntax.
    • Portable: The workspace is an ordinary directory containing Markdown, JSONL, and YAML files that can be backed up or versioned.
    • Indexable: ReMe parses frontmatter, body chunks, and wikilinks to build retrieval indexes and a file graph.
  6. Understand Auto Memory recording and storage

    main

    Auto Memory distills conversations into daily memory cards. It does not store a running chat transcript summary; instead, it extracts useful information such as user preferences, key facts, process decisions, current state, and reusable experiences.

    Data is stored in a hierarchical structure:

    1. Individual Memory Cards: One .md file per conversation, stored in daily/YYYY-MM-DD/<session_id>.md.
    2. Daily Index: A single daily/YYYY-MM-DD.md file that indexes all memory cards for that specific day.
    3. Original Conversations: Raw session data is preserved for verification in session/dialog/<session_id>.jsonl.
  7. Security and Agent Boundaries in Auto Fin

    main

    The Auto Fin wrapper operates with specific security and capability settings:

    • Capabilities: It loads the tushare-data skill and exposes the memory_search job tool.
    • Permissions: It defaults to bypassPermissions. Warning: bypassPermissions is not an operating-system sandbox. You must review the configured project path, workspace, credentials, and network boundaries before deployment.
    • Memory Search: By default, the standalone cookbook does not configure an embedding store, so memory_search uses BM25 recall. Vector and BM25 fusion are only available if an embedding store is explicitly configured.
  8. Understand the ReMe workspace layout

    main

    By default, ReMe creates a .reme/ directory in your current folder to manage state and memory. The structure is as follows:

    • metadata/: Persistent indexes, graphs, catalogs, and related state.
    • session/: Agent sessions and original conversations.
    • resource/: External resources.
    • daily/: Daily notes.
    • digest/: Long-term memory.
  9. Understand ReMe semantic memory chunking

    main

    ReMe uses semantic chunking rather than fixed-length window splitting. While traditional RAG often cuts text at arbitrary token counts (potentially breaking headings, tables, or links), ReMe's MarkdownFileChunker preserves the document skeleton and structural context.

    Key differences in ReMe chunks:

    • Structure: Each chunk includes its heading skeleton (e.g., # Top-level heading followed by ## Current section) so the agent understands the structural position of the fragment.
    • Integrity: The chunker attempts to keep headings, tables, code blocks, and [[wikilinks]] intact.
    • Rules:
      • Frontmatter is parsed separately from the body.
      • A section tree is built from heading levels; the system prefers one complete section per chunk.
      • If a section is too long, it recursively splits subsections.
      • Tables repeat headers when split; code blocks repeat fences; lists are packed by item.
      • As a last resort, it splits greedily by line and appends [Part X/N].

    Non-Markdown files: Use the DefaultFileChunker, which splits by byte size with a small overlap.

  10. Understand the ReMe workflow pattern

    main

    ReMe follows a structured data flow to transform raw information into long-term knowledge:

    1. Ingestion: Conversations and external resources are processed by auto_memory or auto_resource and written to the daily/ directory.
    2. Distillation: The auto_dream pipeline processes daily/ entries to distill them into structured knowledge in digest/{personal,procedure,wiki}/ and generates an interests.yaml file for each day.
    3. Retrieval: Agents use search, node_search, read, traverse, or proactive capabilities to retrieve, associate, and inspect topics within the knowledge graph.
  11. Understand Auto Fin workflow steps

    main

    The Auto Fin job is composed of four main steps:

    StepResponsibilityAgent
    auto_fin_data_stepMaintain news files and resolve the previous trading dayNo
    auto_fin_topic_stepBuild inputs and select related ETFs and current eventsYes
    auto_fin_history_stepOrchestrate historical research and market analysis per ETFYes
    auto_fin_merge_stepValidate results and produce the final Markdown reportYes

    Agents handle semantic judgments (like event similarity), while deterministic code handles source validation and financial calculations (like adjusted returns).

  12. Review the Auto Dream workflow stages

    main

    The auto_dream flow consists of four sequential stages:

    1. Extract (dream_extract_step): Scans changed files in daily/<date>/ and uses an LLM to extract units (long-term memory units for digest/) and topics (proactive interests).
    2. Integrate (dream_integrate_step): An agent processes each unit to integrate it into the digest/ directory. It uses tools like node_search, read, write, and edit to perform actions such as CREATE, CORROBORATE, REFINE, or CORRECT on existing nodes.
    3. Topics (dream_topics_step): Refines topic candidates into a final daily/<date>/interests.yaml file, performing deduplication against previous days.
    4. Finish (dream_finish_step): Updates the file_catalog with successfully processed paths and returns a summary of the run.