fenic

Fenic enables several categories of semantic data operations:

Core AI Capabilities

• Text Classification: Categorizing text (e.g., medical triage) with zero training data.
• Semantic Joins: Matching messy text to clean data (e.g., location names to API data).
• Embedding Similarity: Matching entities using vector embeddings (e.g., job-candidate matching).
• Content Moderation: Detecting multi-violation context in text.
• Smart Filtering: Filtering data based on meaning rather than exact keyword matches.
• Entity Resolution: Identifying the same entities across disparate data sources.

Data Quality & Enrichment

• Fuzzy Matching: AI-verified customer deduplication.
• Transaction Intelligence: Decoding cryptic payment/financial descriptions.
• Legal Analysis: Extracting and scoring risks from contracts.
• SEO Clustering: Grouping keywords by semantic intent.

Productivity & Workflow

• Jinja Templates: Using dynamic AI prompts that adapt to data.
• Document Summarization: Converting long documents into insights.
• Meeting Notes Analysis: Extracting action items from transcripts.
• Email Categorization: Organizing and prioritizing inboxes.
• Smart Data Labeling: Generating training data using AI reasoning.

The Fenic documentation MCP (Model Context Protocol) server is being migrated from an archived repository to a dedicated service directory within the main fenic repository.

Key architectural decisions:

• Location: The service source is hosted in services/docs-mcp within the fenic repository.
• Isolation: It is maintained as an independent uv project. This ensures that deployment-specific dependencies (like Modal or Griffe) do not become runtime dependencies of the core fenic Python package.
• Versioning: The production service at mcp.fenic.ai tracks the latest stable fenic release. Deployments are triggered automatically by the fenic release workflow only after the corresponding package is available on PyPI.
• Data Export: Every stable fenic release exports its API documentation as versioned Parquet artifacts to Hugging Face as part of the release pipeline.

Fenic uses two distinct but complementary mechanisms to manage output token rate limits:

1. Settlement (Deterministic Correction):

   • What it is: A mechanism that corrects the TPM bucket after a completion.
   • How it works: It compares the tokens reserved during the request to the actual tokens used (where actual = completion + thinking for models with thinking tokens).
   • Purpose: It ensures the rate-limit bucket accurately reflects reality. Over-reservations are refunded to the bucket, and under-reservations are debited. This is the primary driver of throughput improvements.

2. Adaptive Estimation (Opportunistic Tightening):

   • What it is: A mechanism that shrinks up-front reservations based on recent history.
   • How it works: It uses a moving window of observed usage to calculate a reservation value (quantile × safety_margin).
   • Purpose: It allows more requests to be admitted into a single burst by not over-reserving tokens that aren't actually being used, thereby increasing 'reservation efficiency'.

Relationship: Settlement is always-on to maintain bucket integrity, while Adaptive Estimation is an optional layer that optimizes how many tokens are reserved before a request is dispatched.

Fenic uses a structured type system defined by Schema and ColumnField objects.

Core Components:

• ColumnField: A frozen pydantic dataclass containing a name and a data_type.
• Schema: A frozen pydantic dataclass containing an ordered list of column_fields.

Supported Data Types:

• Primitives: string, integer, float, double, boolean, date, timestamp.
• Composites: ArrayType, StructField, and StructType.
• Logical Types: EmbeddingType, and tagged strings for markdown, HTML, and JSON.
• Parameterized Types: transcripts and document paths.

Type Mapping (Polars to Fenic):

• Polars Utf8 $\rightarrow$ Fenic string
• Polars Datetime $\rightarrow$ Fenic timestamp (normalized to microsecond UTC)
• Polars List $\rightarrow$ Fenic ArrayType
• Polars Struct $\rightarrow$ Fenic StructType
• Polars Fixed-size Array $\rightarrow$ Fenic EmbeddingType (maps to pl.Array(pl.Float32, dimensions))

Fenic supports two strategies for structured outputs via OpenRouter, depending on model capabilities:

1. Pydantic Structured Outputs (via Response Format): The preferred method when the model supports structured_outputs. Fenic sends the JSON Schema derived from your Pydantic model to constrain the output.
2. Forced Tool Calling (with JSON Schema): Used when native structured outputs are unavailable or when structured_output_strategy="prefer_tools" is set. Fenic registers an output_formatter tool and explicitly forces the model to call it using tool_choice.

Note on Compatibility:

• Forced tool choice is incompatible with manual extended thinking on Anthropic models. For these models, Fenic uses native structured outputs if available, otherwise it fails fast.
• If a model does not support either method and a semantic operation (like semantic.extract) is called, Fenic will fail immediately.

The Fenic MCP server provides two generated tools—Search Content and Search Summary—that support two distinct search modes via the search_mode parameter. This allows you to perform exact substring matches without needing to escape regex metacharacters.

Search Modes

• regex (default): Treats the pattern as a regular expression. This is the backward-compatible default behavior.
• literal: Treats the pattern as a plain substring. This is useful when your search term contains characters like ., *, or ? that you want to match literally.

Tool Behavior

• Search Content: Returns matching rows from a single dataset. Supports paging (limit, offset), ordering (order_by, sort_ascending), and restricting search to specific columns (search_columns).
• Search Summary: Returns the number of matches per dataset across all available datasets. It uses the same search_mode semantics as Search Content.

# Example of how the generated tool signature appears
# (Conceptual representation of the tool's callable signature)

async def search_content(
    pattern: str, 
    search_mode: Literal["regex", "literal"] = "regex", 
    limit: int = 10, 
    offset: int = 0, 
    # ... other parameters
):
    ...

async def search_summary(
    pattern: str, 
    search_mode: Literal["regex", "literal"] = "regex",
    # ... other parameters
):
    ...

The Serde system uses specific patterns for handling complex data:

Complex Nested Structures

When serializing nested objects (like a CaseExpr), use context.serialize_logical_expr_list for collections and context.serialize_logical_expr for single optional nested elements.

Enum Handling

To serialize an enum value, use context.serialize_enum_value(field_constant, value, EnumProtoClass).

Optional Field Handling

During deserialization, use proto.HasField("field_name") to determine if an optional field should be processed or returned as None.

# Complex Nested Structures
@serialize_logical_expr.register
def _serialize_case_expr(expr: CaseExpr, context: SerdeContext) -> LogicalExprProto:
    return LogicalExprProto(
        case=CaseExprProto(
            when_exprs=context.serialize_logical_expr_list(SerdeContext.EXPRS, expr.when_clauses),
            else_expr=context.serialize_logical_expr("else_expr", expr.else_clause) if expr.else_clause else None,
        )
    )

# Enum Handling
@serialize_logical_expr.register
def _serialize_binary_expr(expr: BinaryExpr, context: SerdeContext) -> LogicalExprProto:
    return LogicalExprProto(
        binary=BinaryExprProto(
            left=context.serialize_logical_expr(SerdeContext.LEFT, expr.left),
            right=context.serialize_logical_expr(SerdeContext.RIGHT, expr.right),
            operator=context.serialize_enum_value(SerdeContext.OPERATOR, expr.op, OperatorProto),
        )
    )

# Optional Field Handling
@_deserialize_logical_expr_helper.register
def _deserialize_optional_expr(proto: OptionalExprProto, context: SerdeContext) -> OptionalExpr:
    return OptionalExpr(
        required_field=context.deserialize_logical_expr("required", proto.required_field),
        optional_field=context.deserialize_logical_expr("optional", proto.optional_field)
                      if proto.HasField("optional_field") else None,
    )

Fenic supports logical string-backed types that provide specialized functionality without changing the underlying physical storage.

• Behavior: When a column is defined with JsonType or MarkdownType in a Schema, df.schema.column_fields will reflect these logical types. However, calling df.to_polars() will return a Polars DataFrame where these columns are stored as physical strings.
• Usage: You can immediately use specialized functions on these columns without manual casting. For example, a column typed as JsonType can be passed directly to json.jq() functions, and a MarkdownType column can be passed to markdown.generate_toc().

This allows the logical plan to understand the semantic meaning of the data while maintaining compatibility with standard string-processing engines.

# Creating a JSON-typed DataFrame
schema = Schema([ColumnField("json_col", JsonType)])
df = session.create_dataframe(data=my_data, schema=schema)

# You can use JSON functions directly without .cast(JsonType)
result = df.json.jq("json_col", ".user.name")

The PDF Parsing Evaluation Harness is composed of four main modules that orchestrate the lifecycle of parsing and grading PDF documents using LLMs and Fenic:

1. eval_models.py: The main orchestration harness using the PDFEvalTestHarness class. It coordinates processing and grading across multiple models, manages the Fenic session lifecycle, and aggregates results.
2. pdf_processor.py: The parsing module. It uses semantic.parse_pdf() to perform LLM-based parsing, persists the results to Fenic tables, and writes individual Markdown files.
3. grade_md.py: The evaluation module. It orchestrates text and structure fidelity grading, computes weighted scores, and saves results to the eval_results table.
4. grade_md_structure.py: The structure analysis module. It extracts structural elements from PDFs (via PyMuPDF) and parses Markdown (via markdown-it) to calculate precision, recall, and F1 scores.

Data Flow: Input PDFs $\rightarrow$ pdf_processor.py $\rightarrow$ Markdown Files + Fenic Table $\rightarrow$ grade_md.py $\rightarrow$ Scores in eval_results table $\rightarrow$ eval_models.py (retrieval and display).