SirixDB Documentation

When designing your application with Sirix, note the following workload characteristics:

The Projection Index Storage Redesign is being implemented in phases (P0 through P8). As of 2026-07-20, the following status applies:

• P0/P1 (Page Layer): Done. Includes ProjectionSegmentPage (PageKind 18), HOT side-map commit chains, and split-time ref routing.
• P2 (Descriptor + Segment Codec): Done. Includes PIXD descriptor, PIXS segments, and XXH3 integrity hashes.
• P3 (Storage Rewrite): Done. Includes ProjectionIndexHOTStorage with putLeaf/putEncodedLeaf, tombstone support, and resetTree migration.
• P4 (Builder + Maintenance): Done. Includes streaming builds and per-segment maintenance writes.
• P5 (Query Integration): Parity done. Serving/gate/differential suites are green; columnar Handle/kernel restructure is deferred.
• P6 (Numeric Double + ALP): Done. Supports NUMERIC_DOUBLE via sortable-bits transform and ALP compression.
• P7 (FSST Dictionaries): Done. Uses mode-byte DICT segments with deterministic training.
• P8 (Migration + Docs): Done. ProjectionIndexMetadata.VERSION = 0 and rebuild-on-open logic implemented.

The optimizer implementation is located in bundles/sirix-query/src/main/java/io/sirix/query/. Key components include:

• Entry Points: SirixCompileChain.java (creates optimizer + translator) and SirixQueryContext.java (handles statistics invalidation).
• Optimizer Pipeline: SirixOptimizer.java orchestrates a 10-stage pipeline (including JQGM rewrites, cost analysis, join reordering via DPhyp/GOO, and SIMD/Vectorized detection) with a 50ms circuit breaker.
• Join Ordering: AdaptiveJoinOrderOptimizer.java uses DPhyp for $\le 20$ relations and GOO for $> 20$ relations.
• Search Space: Mesh.java and EquivalenceClass.java manage alternative plan groupings.
• Statistics & Cost Model: Includes SelectivityEstimator.java, CardinalityEstimator.java, and Histogram.java (supporting equi-width, equi-depth, and MCV).
• Physical Operators: IndexExpr.java (reads from CAS/PATH/NAME indexes) and VectorizedPipelineExpr.java (SIMD batch processing).
• Query Inspection: sdb:explain() function and QueryPlan.java for programmatic plan inspection.

bundles/sirix-query/src/main/java/io/sirix/query/
├── SirixCompileChain.java              ← Entry point: creates optimizer + translator
├── SirixQueryContext.java              ← Post-commit statistics invalidation
├── compiler/
│   ├── XQExt.java                      ← Extension AST node types (IndexExpr, VectorizedPipelineExpr)
│   ├── translator/
│   │   ├── SirixTranslator.java        ← Converts optimized AST → physical operators
│   │   ├── SirixPipelineStrategy.java  ← Intersection join: forces hash mode on TableJoin
│   │   └── ...
│   ├── expression/
│   │   ├── IndexExpr.java              ← Physical operator: reads from CAS/PATH/NAME index
│   │   └── VectorizedPipelineExpr.java ← Physical operator: SIMD batch scan-filter-project
│   ├── optimizer/
│   │   ├── SirixOptimizer.java         ← Orchestrates 10-stage pipeline with 50ms circuit breaker
│   │   ├── PlanCache.java              ← LRU cache: queryText+schemaVersion → optimized AST
│   │   ├── CardinalityTracker.java     ← Detects estimate-vs-actual drift, invalidates stale plans
│   │   └── ...
│   └── ...
└── function/sdb/explain/
    ├── Explain.java                    ← sdb:explain() XQuery function
    ├── QueryPlan.java                  ← Programmatic plan inspection API
    └── QueryPlanSerializer.java        ← Converts AST → human-readable JSON

A projection index is a persistent, incrementally-maintained columnar projection of selected fields from a JSON record set. It allows SirixDB to serve OLAP-style analytical queries (such as sum, avg, count, group-by, and count-distinct) at columnar-engine speeds without scanning the full JSON document tree.

Key properties:

• Versioned: It lives inside SirixDB's copy-on-write page tree, meaning every commit produces a new queryable snapshot. Time-travel analytics can run on the same kernels as current-state queries.
• Automatically Maintained: A change listener patches the columns at every commit (rebuilding leaves, appending tails, or dropping rows). No manual refresh is required.
• Fail-Closed: If the projection cannot prove it can answer a query exactly (due to type provenance, presence, or staleness), the query silently falls back to the generic document-scan pipeline to ensure correctness.

For queries too large for exhaustive DP, the GOO-DP strategy is used. It works in two distinct phases:

Phase 1: Greedy Operator Ordering

The optimizer builds an initial bushy plan by iteratively selecting the pair of available relations or sub-plans that produces the minimum join result cardinality. If the join graph is disconnected, it forces a Cartesian product between the two smallest remaining components to minimize cross-product costs.

Phase 2: DP Refinement

The optimizer identifies 'boundary subtrees'—subtrees that are small enough to be re-optimized (size $\le$ maxSubproblemSize) but are part of a larger, more expensive structure.

Refinement follows these steps:

1. Collect Candidates: Traverse the join tree to find subtrees within the size limit.
2. Prioritize: Sort candidates by descending estimated cost so the most expensive subtrees are addressed first.
3. Re-optimize: Replace the greedy subtree with a plan generated by exact DP (dpHyp).
4. Budget Management: Each refinement consumes a portion of the CC_BUDGET, estimated by the number of DP table entries (up to $3^n$ for $n$ relations).

Projection indexes use a ProjectionIndexChangeListener to hook into every commit of a resource. The maintenance process follows a specific lifecycle to handle modified nodes:

1. Attribute dirty nodes: Identify which records under the projection root were modified.
2. Read fence chunks: Reassemble the zone map by reading fence chunks.
3. Merge dirty keys: Perform a two-pointer merge of dirty keys against per-leaf fence ranges.
4. Re-extract and Re-encode: Re-extract touched leaves from the document, re-encode them, and use hash-comparison to write only changed segments.
5. Rewrite: Update slot 0 (the shape) and write only the changed fence chunks and segments.

Key Behaviors:

• Appends: Because record keys are monotone, new records always land at the tail and classify cleanly.
• Non-blocking: The maintenance ladder never blocks a commit. If maintenance fails (e.g., due to maxIncrementalRecords being exceeded), the system falls back to a rebuildFully() which may result in a stale tombstone over slot 0, degrading queries until the next manual rebuild or first-use rebuild.
• Sharing: The rebuild process uses hash-compare no-op writes to share unchanged leaf segments, softening the performance impact of large updates.