repository·main·Indexed 20 days ago
https://github.com/aryn-ai/sycamoreAn open-source, AI-powered document processing engine for ETL, RAG, and analytics on unstructured data. Sycamore uses Aryn DocParse to partition complex documents and provides a scalable DocSet abstraction for transforming and loading data into vector databases. It includes integrations for OpenSearch, Jupyter, and BigQuery, as well as a Remote Processor Service for search post-processing.
The sycamore.query package provides the tools necessary to construct sophisticated pipelines driven by Large Language Model (LLM) queries. It is organized into three primary functional areas:
sycamore.query.client): The entry point for interacting with the query system.sycamore.query.planner): Responsible for interpreting queries and determining the necessary execution steps.sycamore.query.logical_plan): Defines the structured representation of the operations required to fulfill a query.DocSet is the core abstraction in the Sycamore framework used for scalable and reliable document processing. It allows you to transform, manipulate, and enrich unstructured documents using a functional programming approach. DocSets encapsulate scalable data processing techniques, removing the heavy lifting required to reliably load document chunks into downstream databases.Sycamore is a document processing engine designed for complex unstructured data (documents, presentations, transcripts, embedded tables, etc.).
It uses a declarative dataflow abstraction called a DocSet to manipulate collections of unstructured documents. This abstraction is similar in style to Apache Spark or Pandas but is optimized for document collections.
Core Capabilities:
Execution operations in Sycamore are categorized into three primary types:
A Sycamore Query plan is an instance of the LogicalPlan class. It represents a tree of operators (nodes) that process data. Operators can be standard data-processing steps (like TopK, count, sort, group-by) or LLM-powered steps (like LlmFilter or SummarizeData).
Users can interact with plans in two ways:
SycamoreQueryClient.query() to let an LLM generate a plan from natural language, or SycamoreQueryClient.generate_plan() to inspect the plan without running it.LogicalPlan directly in code and execute it using SycamoreQueryClient.run_plan().# Example of what a generated plan structure looks like
{
"nodes": {
"0": QueryDatabase(
node_id=0,
description="Get all the incident reports with substantial aircraft damage",
input=None,
index="const_ntsb",
query={"match": {"properties.entity.aircraftDamage": "Substantial"}}
),
"1": TopK(
node_id=1,
description="Get the breakdown of aircraft types",
input=[0],
field="properties.entity.aircraft",
primary_field="properties.entity.accidentNumber",
K=100,
descending=False)
}
}The Embed transform generates embeddings for your Documents or Elements and stores them in a special embedding property on each document. To use it, call the .embed(embedder) method on a DocSet, passing in an embedder object that encapsulates a specific embedding model and its parameters.
embedded_doc_set = docset.embed(embedder)The ingestion pipeline consists of two main stages: crawling and importing.
Note: The Importer supports Generative AI User Defined Functions (UDFs) with various LLMs and vector embedding models. It includes error handling to manage Out-of-Memory (OOM) issues and prevent specific files from failing the entire Sycamore script.
.partition() or .embed()) does not immediately execute the work. The transformations are only triggered when an action requires the data, such as calling .show() to inspect results or .write() to save to a target. This allows Sycamore to optimize the execution plan before running the workload on the Ray backend.The merge transform is used to combine individual elements into larger 'chunks' (a process also known as 'chunking'). To use it, you must provide a merger argument to the docset.merge() method. The merger defines the logic for which elements should be combined and how they should be merged. For detailed implementation details, refer to the merge_elements API documentation.
# General pattern
merged_docset = docset.merge(merger=merger_instance)The Remote Processor Service uses two levels of abstraction to manage search post-processing:
The Workflow:
OpenSearch uses a Remote Search Processor (a plugin) to make an RPC call to the service. This RPC call targets a specific Pipeline endpoint in the service. The service then executes the string of Processors defined in that pipeline.
Essentially, you have an OpenSearch search pipeline that triggers a remote search pipeline.
Sycamore extends OpenSearch capabilities by providing custom search processors via a remote-processor. Instead of running locally within OpenSearch, these processors make a network call to a Sycamore service hosting the logic. This allows Sycamore to provide advanced processing features like result de-duplication and debugging tools that are not native to OpenSearch.
Available Sycamore remote processors include:
dedup: De-duplicates search results based on similarity.debug: Prints the search response to stdout, which is useful for inspecting query behavior.