DOCSEARCHbyAlgolia
Submit Docs

xorq

repository·main·Indexed 19 days ago

https://github.com/xorq-labs/xorq

An executable memory system for tabular data that transforms ephemeral agent-generated scripts (pandas, sklearn, etc.) into durable, composable, and reproducible artifacts stored in a git-native catalog. It supports multi-engine pipelines with caching, lineage, and Arrow Flight for distributed computation via gRPC with zero-copy semantics.

Tokens
34.6K
Snippets
94
Records
161
Agent score
68%

What's inside xorq

  1. Distinguish between Git remotes and Annex special remotes

    main

    In a Catalog, there is a distinction between the Git remote and the Annex special remote:

    1. Git remote: An entry in repo.remotes (e.g., github.com/org/catalog.git) used by push(), pull(), fetch(), and sync(). The Catalog supports at most one of these.
    2. Annex special remote: The storage backend for actual content (e.g., s3://bucket or a local directory) configured via RemoteConfig and stored in remote.log. This remains singular and is independent of the number of git remotes.

  2. Understand the Xorq Catalog structure

    main

    The catalog is a git repository organized into three main directories:

    • aliases/: Contains symlinks to entries, allowing you to reference them by name.
    • entries/: Contains the actual zipped build artifacts.
    • metadata/: Contains YAML sidecar files (*_metadata.yaml) for each entry, enabling fast discovery via grep or file operations.

    This structure allows agents to discover pipelines using simple filesystem commands (e.g., grep for specific columns or engines) without needing a running service.

  3. Using Arrow Flight for distributed computation

    main

    Xorq leverages Arrow Flight to move columnar data between processes over gRPC with zero-copy semantics. This is ideal for running heavy computations (like Python models) in separate processes or exposing transformations as network services.

    A key feature is the do_exchange RPC, which allows for bidirectional streaming of input and output batches, preventing the need to fully materialize large datasets on either side.

    Example Flight Use Cases

    • Serving Models: flight_serve_model.py serves TF-IDF model transformations.
    • Streaming Exchange: flight_exchange_example.py demonstrates an iterative split-train exchanger.
    • UDTFs: flight_udtf_example.py uses a User-Defined Table Function to fetch data via live API calls.
    • Concurrent Servers: duckdb_flight_example.py provides a concurrent Flight server with DuckDB support.

  4. How deferred writes work with TeeNode

    main

    In Xorq, writes can be performed as a side effect of expression execution using a TeeNode. This allows an expression to write its rows to a target while continuing to stream data onward.

    Key characteristics of this model:

    • Side Effect Only: The write does not change the value or result of the expression itself; it is a pass-through node.
    • Cache Neutrality: The TeeNode is hash-neutral. This means the cache hash ignores the tee, but the build hash reflects the write-through.
    • Cache Respecting: If a cache hit occurs, the TeeNode is pruned before execution. No write will be triggered on a cache hit, regardless of the transport method used.
    • Streaming Invariant: Every write is a streaming operation. A WriteThrough consumer writes each batch as a side effect and then yields that batch onward. There are no terminal sink nodes in this model; 'terminal' behavior is achieved through composition.

  5. Manage Python versions with the `--python` flag

    main

    The pipeline uses the --python <version> flag with both uv build and uv tool run to ensure consistency between the build and execution stages.

    Version Selection Logic: When a version is not explicitly provided, the system uses resolve_python_version to read the requires-python specifier from pyproject.toml (e.g., >=3.10) and selects the highest acceptable minor version within a tested range (typically 3.8 through 3.13).

    Rationale for highest version:

    • Forward compatibility: Catches issues with newer interpreters early.
    • Ecosystem support: Newer versions often have better pre-built wheel availability on PyPI.
    • Determinism: Makes the Python version a function of project metadata rather than the host system.

  6. Understand the difference between Build Hash and Cache Hash

    main

    Xorq uses two distinct hashes derived from the same expression tokenizer to manage artifacts and performance:

    1. Build Hash (get_expr_hash in provenance_utils.py): Identifies the unique build artifact. It is used by the catalog to answer "has this pipeline been built?". Invariant: Every operation in the DAG must participate in the build hash to prevent collisions between structurally different pipelines.
    2. Cache Hash (expr.ls.tokenized): Determines cache hits for a CachedNode. It decides whether to recompute an expression or use a cached result.

    Key Rule: While every op must be in the build hash, an op may be cache-hash-neutral (stripped from the cache hash) only if it is a pure side-effect (e.g., writing, tagging, metadata) that does not change the input/output rows or schema.

    # Concept Summary
- Build Hash: Identity of "what was built" (must be unique per DAG).
- Cache Hash: Identity of "what to recompute" (can ignore side-effects).
- Rule: Side-effect-only ops (like `Tag` or `TeeNode`) can be cache-neutral, but must remain in the build hash.

  7. Understand WriteThrough and Transport terminology

    main

    When working with deferred writes in Xorq, it is important to distinguish between the consumer logic and the delivery mechanism:

    • WriteThrough: The consumer abstraction (ABC) that implements the side-effect behavior. It defines the write_through method used to write each batch and yield it onward.
    • Transport: The mechanism defining how batches reach the target.
      • Client-side Arrow generator (Phase 1): The default, portable transport that round-trips the parent through Arrow via a client-side streaming generator.
      • In-engine transport: An optimized method (e.g., CTAS / writable CTE) used when the parent expression and the write target share the same backend.

  8. Understand Xorq documentation types

    main

    Xorq documentation is organized into five distinct types to help you find the right level of detail for your needs:

    • Quickstart: Designed to get you to a first successful build or run as fast as possible (e.g., "Get started with Xorq in 5 minutes").
    • Tutorial: A step-by-step, learning-focused walkthrough for beginners that takes you through a meaningful project.
    • How-to guide: Task-based instructions for specific goals aimed at users with existing knowledge (e.g., "How to serve a pipeline with Xorq and DuckDB").
    • Reference: Detailed technical specifications for APIs, CLI commands, configuration keys, and schemas.
    • Concept: Explanations of core ideas, architecture, and terminology (e.g., "Deferred execution", "Profiles").

  9. Canonicalization process for in-memory table data

    main

    When generating a canonical digest for table columns, xorq follows a strict, order-dependent transformation pipeline to ensure stability and prevent overflows:

    1. Proxy Data Access: Uses stored proxy data rather than re-executing a backend.
    2. Type Widening & Rewriting: For each column, it recursively decodes dictionary encoding, widens int32-offset var-length types to large_* types, and rewrites string_view/binary_view to large_string/large_binary using a single chunk-wise cast.
    3. Compaction: Performs combine_chunks/concat_arrays to create a contiguous array.
    4. Metadata Stripping: Produces a metadata-free single-column RecordBatch.
    5. Hashing: Computes the xxh128 hash of the resulting IPC bytes.

    Note: The type cast must occur before compaction to prevent overflow in columns where a >2 GiB string column might exceed the addressing limits of an int32-offset array.

  10. Distinction between Expressions and Scripts in xorq

    main

    When working with xorq examples, it is important to distinguish between Expressions and Scripts to know how to execute them correctly.

    • Expressions: These are defined by a top-level expr variable. They represent a declarative computation graph. When run via ./example_run.sh, xorq compiles the expression into a versioned YAML manifest before execution. This is the standard way to use xorq for data processing workflows.
    • Scripts: These are standard Python files that perform imperative tasks, such as starting a Flight server, running CLI subcommands, or managing external processes. These cannot be compiled as expressions and must be executed directly using python <file>.py.

  11. Understand Catalog remote limitations and errors

    main

    The Catalog API is designed to work with either zero remotes (local-only mode) or exactly one git remote.

    Supported Configurations

    • Zero remotes: Used for local-only workflows (e.g., development, testing, or before a remote is wired up). In this mode, push(), pull(), fetch(), and sync() are treated as no-ops and do not raise errors.
    • One remote: The standard supported configuration for versioned build artifacts.

    Unsupported Configurations

    • Two or more git remotes: If the underlying git repository has multiple remotes with a fetch refspec, the Catalog will refuse to operate.

    If you attempt to call push(), pull(), fetch(), or sync() on a repository with multiple remotes, the library will raise a CatalogConfigurationError and name the remotes found.

  12. Understand the difference between `.tee()` and `into_backend`

    main

    While both involve writing data to a backend, they serve different purposes regarding ownership and lifecycle:

    • .tee() (Deferred Write): Produces a persistent, user-named, user-owned target. The target is never temporary and is not automatically dropped; teardown must be explicit. It leaves the original data flowing through the pipeline.
    • into_backend (RemoteTable): Produces an anonymous, ephemeral table that is automatically dropped during the clean_up phase. It is used for temporary intermediate storage.