DOCSEARCHbyAlgolia
Submit Docs

Arcana: Embeddable RAG Library for Elixir and Phoenix

repository·main·Indexed 43 days ago

https://github.com/georgeguimaraes/arcana

An embeddable Retrieval-Augmented Generation (RAG) library for Elixir and Phoenix that integrates vector search, knowledge graphs, and LLM-driven retrieval into Postgres/Ecto applications. It offers three modes of operation: Advanced RAG for standard use cases, Modular RAG via Arcana.Pipeline for fine-grained control, and Agentic RAG via Arcana.Loop for LLM-driven tool selection. Features include hybrid search, cross-encoder reranking, and faithfulness evaluation.

Tokens
47.9K
Snippets
153
Records
204
Agent score
55%

What's inside Arcana

  1. Overview of Dashboard Pages

    main

    The Arcana dashboard provides several functional areas for managing your RAG setup:

    • Documents: Browse, chunk, ingest text, or upload files (.txt, .md, .pdf).
    • Ask: Test retrieval using three modes:
      • Advanced: Automated hybrid search, reranking, and LLM answering.
      • Pipeline: Modular RAG where you manually toggle steps (gate, rewrite, expand, etc.).
      • Loop: Agentic RAG where a controller LLM uses tools.
    • Search: Test queries against vector, keyword, or hybrid modes and view similarity scores.
    • Collections: Create, edit, or delete document collections.
    • Evaluation: Run and compare evaluation metrics like MRR, Precision, and Recall.
    • Maintenance: Rebuild embeddings, clean up orphan chunks, or perform database operations.
    • Info: View configuration, embedding models, and statistics.

  2. Review the Arcana roadmap and planned features

    main

    Arcana is actively evolving. Current capabilities include Hybrid search (vector + full-text with RRF), modular pipelines via Arcana.Pipeline, GraphRAG, and agentic loops via Arcana.Loop.

    Planned features include:

    • HyDE (Hypothetical Document Embeddings)
    • Async ingestion via Oban
    • Additional vector backends (TurboPuffer, ChromaDB)

  3. Overview of Arcana Search Modes

    main

    Arcana provides three distinct search modes to balance semantic meaning and exact keyword matching. The implementation of these modes depends on whether you are using the in-memory backend or the PgVector (PostgreSQL) backend.

    ModePurposeMemory BackendPgVector Backend
    :vectorFind similar meaningHNSWLib cosine similaritypgvector HNSW index
    :keywordFind keyword matchesTF-IDF-like scoringPostgreSQL tsvector
    :hybridCombine bothTwo queries + RRFSingle-query with weights

    When to use each mode

    • :vector: Use for conceptual questions (e.g., "How does X work?") or finding related concepts.
    • :keyword: Use for exact terms, specific names, or code snippets (e.g., API/function lookups).
    • :hybrid: Use for general search when the query type is unknown, as it combines the strengths of both modes.

  4. Understand the Faithfulness metric

    main

    The Faithfulness metric measures whether a generated answer is grounded in the retrieved context chunks. It uses an LLM-as-judge approach to score the relationship between the answer and the source material.

    Scale (0-10):

    • 0: Completely unfaithful, hallucinated, or contradicts the context.
    • 5: Partially supported; some claims lack grounding.
    • 10: Fully faithful; every claim is supported by the context.

    When running an evaluation with evaluate_answers: true, the aggregate average faithfulness score is included in run.metrics, and individual scores/reasoning are included in the per-case results.

  5. Understand the transition from Arcana.Agent to Arcana.Pipeline

    main

    The module Arcana.Agent has been renamed to Arcana.Pipeline to better reflect its modular RAG capabilities.

    • Compatibility: Arcana.Agent remains available as a deprecated alias. Existing calls to Arcana.Agent.search/2 will continue to work but will emit a warning.
    • Recommendation: New implementations should use Arcana.Pipeline directly.

  6. Perform Vector Search

    main

    Vector search finds semantically similar content using cosine similarity.

    • Memory Backend: Uses HNSWLib for approximate nearest neighbor search. The score is calculated as 1.0 - cosine_distance.
    • PgVector Backend: Uses the PostgreSQL pgvector extension with HNSW indexing. It uses the <=> operator to compute cosine distance efficiently.

  7. Restrict dashboard access with Collection Scoping

    main

    By default, the dashboard has superuser access to all collections. To restrict a user to specific collections, use the collection: option. This option takes a {module, function} tuple. The function receives the %Plug.Conn{} and must return one of the following:

    • :all: Access to all collections.
    • "collection_name": Access to a single collection.
    • ["name1", "name2"]: Access to a specific list of collections.
    • []: Access to nothing.

    Security Note: The scope is snapshotted into the signed LiveView session at mount time. To force a permission update, you must disconnect the user's socket (e.g., via Endpoint.broadcast) to trigger a full page reload/re-mount.

    # Router configuration
arcana_dashboard "/arcana",
  collection: {MyAppWeb.ArcanaAccess, :allowed_collections}

# Implementation
defmodule MyAppWeb.ArcanaAccess do
  def allowed_collections(conn) do
    case conn.assigns.current_user do
      %{admin: true} -> :all
      %{tenant: tenant} -> ["#{tenant}-docs", "#{tenant}-tickets"]
      _ -> []
    end
  end
end

  8. Ordering and concurrency caveats in ingestion

    main

    Arcana uses an advisory lock to handle races, but there is a specific behavior regarding ordering:

    If two jobs for the same identity execute concurrently (i.e., the second job starts before the first one finishes), Arcana follows a first-to-complete-wins policy, not necessarily a latest-wins policy.

    If your application requires strict latest-wins behavior (where the most recently enqueued content must always be the one that persists), you should:

    1. Run the ingestion queue with limit: 1,
    2. OR have the worker return {:snooze, n} if it detects an earlier job for the same identity is still executing.

  9. Reference: Loop and Tool Call Events

    main

    The Arcana.Loop module emits a span for the entire agent loop run, plus discrete events for every tool call. This is useful for rendering live traces (like an "Agent thinking" panel).

    EventMetadata
    [:arcana, :loop, :*]question, max_iterations, tool_count, iterations, terminated_by
    [:arcana, :loop, :tool_call]tool, args, iteration, summary, returned_chunk_ids
    [:arcana, :loop, :ground, :*]chunk_count, answer_present

    Loop Termination States (:terminated_by):

    • :answered
    • :gave_up
    • :max_iterations
    • :error

    Tool Call Metadata: The [:arcana, :loop, :tool_call] event metadata mirrors a single tool_history entry exactly. Measurements include count: 1 and history_size: n.

    # Example: Attaching a custom handler to trace tool calls
:telemetry.attach(
  "loop-trace",
  [:arcana, :loop, :tool_call],
  fn _event, _measurements, metadata, _config ->
    IO.puts("[#{metadata.iteration}] #{metadata.tool}: #{inspect(metadata.args)}")
  end,
  nil
)

  10. Use Arcana.Pipeline for Modular RAG

    main

    Use Arcana.Pipeline when you need explicit control over the order and behavior of RAG steps. This is ideal for non-default sequences (e.g., rewriting before decomposing), inspecting intermediate states like ctx.sub_questions or ctx.reason_iterations, or plugging in custom behaviors like your own searcher or reranker.

    If you want the default behavior provided by Arcana.search/2 or Arcana.ask/2, use those instead. If the sequence of searches needs to be decided dynamically by an LLM, use Arcana.Loop.

    alias Arcana.Pipeline

ctx = 
  Pipeline.new("Compare Elixir and Erlang")
  |> Pipeline.gate()        # Decide if retrieval is needed
  |> Pipeline.rewrite()     # Clean up conversational input
  |> Pipeline.expand()      # Expand query with synonyms
  |> Pipeline.decompose()   # Break into sub-questions
  |> Pipeline.search()      # Search for each sub-question
  |> Pipeline.reason()      # Multi-hop: search again if needed
  |> Pipeline.rerank()      # Re-rank results
  |> Pipeline.answer()      # Generate final answer
  |> Pipeline.ground()      # Detect hallucinations

ctx.answer

  11. Understand the difference between Arcana.Pipeline and Arcana.Loop

    main

    Arcana distinguishes between two primary RAG patterns to align with industry taxonomy:

    1. Arcana.Pipeline (formerly Arcana.Agent): This is a Modular RAG pattern. It is a composed, pluggable sequence of steps (e.g., search, rerank, rewrite) where the developer defines the flow. The LLM does not drive the execution logic; instead, you compose the pipeline using specific step modules.

    2. Arcana.Loop: This is an Agentic RAG pattern. It implements an LLM-driven loop where the model iteratively picks tools and makes decisions based on the intermediate state of the conversation or task.

    Use Arcana.Pipeline when you want a predictable, composed sequence of RAG steps, and use Arcana.Loop when you need an autonomous agent that can reason and use tools dynamically.

  12. Perform Hybrid Search

    main

    Hybrid search combines vector and keyword results. The implementation strategy depends on your backend:

    PgVector Backend (Single-Query Hybrid)

    Uses a single SQL query that combines vector and keyword scores using min-max scaling for the keyword component. This ensures that items ranking moderately in both lists are not missed.

    Memory Backend (Reciprocal Rank Fusion)

    Uses Reciprocal Rank Fusion (RRF) to combine results from two separate queries. Instead of raw scores, RRF uses the rank position of an item to calculate a score, preventing one method's scale from biasing the results. The formula is 1.0 / (k + rank), where k defaults to 60.