ragnar

ragnar is an R package designed to implement Retrieval-Augmented Generation (RAG) workflows. It provides a complete solution for building RAG pipelines with sensible defaults while allowing users to inspect intermediate steps to avoid a "black box" experience.

A typical ragnar workflow follows these stages:

1. Document Processing: Preparing markdown or HTML files.
2. Text Chunking: Dividing documents into smaller, semantically meaningful pieces.
3. Context Augmentation (Optional): Adding document-level context to individual chunks to improve retrieval.
4. Embedding: Computing vector representations for chunks (typically via ellmer).
5. Storage: Saving chunks and embeddings (typically in duckdb).
6. Retrieval: Finding relevant chunks using cosine similarity and BM25 ranking.
7. Re-ranking (Optional): Using specialized models to refine the order of retrieved chunks.
8. Prompt Generation: Injecting retrieved context into an LLM prompt.

ragnar implements Retrieval-Augmented Generation (RAG) through a multi-step pipeline designed for transparency and control. The workflow consists of:

1. Document Processing: Converting files or URLs to Markdown using MarkItDown via read_as_markdown().
2. Text Chunking: Dividing documents into semantic chunks using markdown_chunk(). This step can optionally include Context Augmentation, where markdown_chunk() automatically associates chunks with their parent headings/subheadings.
3. Embedding: Computing vector representations for chunks. While specific functions like embed_openai() exist, they are typically passed as a callback to ragnar_store_create() and called automatically during insertion and retrieval.
4. Storage: Saving processed data in a DuckDB-optimized format using ragnar_store_create(), ragnar_store_connect(), and ragnar_store_insert().
5. Retrieval: Finding relevant chunks using ragnar_retrieve() (which combines vector similarity search vss and bm25 text search) or specific methods like ragnar_retrieve_vss() and ragnar_retrieve_bm25().
6. Chat Augmentation: Integrating the store into an ellmer::Chat object using ragnar_register_tool_retrieve() to allow LLMs to query the knowledge base on-demand.

This example demonstrates the full pipeline: finding links, reading markdown, chunking, creating a store with OpenAI embeddings, ingesting data, and retrieving relevant chunks.

library(ragnar)

# 1. Setup: Find links and define store
base_url <- "https://r4ds.hadley.nz"
pages <- ragnar_find_links(base_url, url_filter = \(url) {
  url[startsWith(url, "https://r4ds.hadley.nz")]
})

store_location <- "r4ds.ragnar.duckdb"

# 2. Create store with embedding function
store <- ragnar_store_create(
  store_location,
  embed = \(x) ragnar::embed_openai(x, model = "text-embedding-3-small")
)

# 3. Ingest pages
for (page in pages) {
  message("ingesting: ", page)
  chunks <- page |> read_as_markdown() |> markdown_chunk()
  ragnar_store_insert(store, chunks)
}

# 4. Build index
ragnar_store_build_index(store)

# 5. Retrieve
store <- ragnar_store_connect(store_location, read_only = TRUE)
text <- "How can I subset a dataframe with a logical vector?"
relevant_chunks <- ragnar_retrieve(store, text)
print(relevant_chunks)

The knowledge store is powered by DuckDB. Use these functions to manage the lifecycle of your data:

• ragnar_store_create(location, embed): Creates a new store at location. The embed argument must be a function that takes text and returns embeddings.
• ragnar_store_connect(location, read_only = FALSE): Connects to an existing store.
• ragnar_store_insert(store, chunks): Inserts chunked data into the store.
• ragnar_store_build_index(store): Builds the necessary search indices for the store.
• ragnar_retrieve(store, text): High-level retrieval that performs both vss (vector similarity) and bm25 (text search) and de-overlaps the results.
• ragnar_retrieve_vss(store, text): Retrieves using the DuckDB vss extension.
• ragnar_retrieve_bm25(store, text): Retrieves using the DuckDB full-text search extension.
• chunks_deoverlap(chunks): Consolidates retrieved chunks that overlap.