datapizza-ai

The FastEmbedder provides the following capabilities:

• Efficient Sparse Embeddings: Uses FastEmbed for high-performance sparse text embeddings.
• Local Execution: Models run locally on your machine; no external API calls are required.
• Configurable Caching: Supports a configurable model caching directory.
• Custom Naming: Allows defining a custom embedding_name.
• Memory Efficiency: Uses a sparse embedding format to reduce memory footprint.
• Async Support: Provides both synchronous and asynchronous embedding methods.

The GoogleEmbedder provides the following capabilities:

• Model Support: Works with Google's Gemini embedding models.
• Batching: Handles both single text strings and lists of texts for batch embedding.
• Async Support: Provides a_embed() for asynchronous workflows.
• Management: Handles automatic client initialization and management using Google's Generative AI SDK.

Treebuilders are pipeline components designed to construct hierarchical tree structures composed of Node objects from various types of content. They transform flat or unstructured data (such as articles, reports, or manuals) into organized, nested representations. This facilitates better semantic processing and understanding of the content.

Key capabilities include:

• Semantic Organization: Uses language models to understand content structure.
• Metadata Extraction: Adds tagging and metadata during the creation process.
• Configurability: Supports configurable tree depth and structure rules.
• Processing Modes: Supports both synchronous and asynchronous processing.

The TextParser module converts plain text into a structured hierarchy of nodes. It splits text into paragraphs (based on double newlines) and then breaks those paragraphs into sentences using regex patterns.

This creates a three-level hierarchy:

1. DOCUMENT: The root container.
2. PARAGRAPH: Intermediate nodes representing paragraphs.
3. SENTENCE: Leaf nodes containing the actual text content.

Each paragraph and sentence node includes index metadata to preserve the original structure.

from datapizza.modules.parsers.text_parser import TextParser

parser = TextParser()
document_node = parser.parse("Your text content here", metadata={"source": "example"})

The NodeSplitter provides several key capabilities for document processing:

• Maintains Node structure: Preserves the object structure and hierarchy of the original document.
• Metadata Preservation: Keeps metadata from the original nodes attached to the resulting chunks.
• Boundary Respect: Attempts to respect existing node boundaries during the splitting process.
• Flexible Chunking: Supports both structure-preserving and flattened chunking modes.
• Nested Relationship Handling: Intelligently manages nested node relationships.

The Memory class is used to store and manage the sequence of exchanges in a conversation. This enables the AI to reference previous turns.

To maintain an accurate history, use the following pattern:

1. Retrieve Context: Pass the Memory object to client.invoke(..., memory=memory) to include history in the prompt.
2. Record User Input: Use memory.add_turn(TextBlock(content=user_input), role=ROLE.USER) to save the user's message.
3. Record Assistant Response: Use memory.add_turn(response.content, role=ROLE.ASSISTANT) to save the AI's reply.

Note: User turns typically require wrapping the content in a TextBlock object.

Parsers are pipeline components responsible for converting documents into structured, hierarchical Node representations. They extract text, layout information, and metadata from various document formats to create tree-like data structures.

Key Requirement: Every parser must return a Node object. If you implement a custom parser that returns a different type (such as raw plain text), you must use a TreeBuilder to convert that output into a Node structure before proceeding with the pipeline.