DOCSEARCHbyAlgolia
Submit Docs

DataJunction Semantic Layer

repository·main·Indexed 41 days ago

https://github.com/datajunction/dj

An open-source metrics platform that serves as a semantic layer over physical data warehouses. It enables the definition of metrics and data models using SQL and automated query generation via a dependency graph. The platform includes a Python client with DJClient for read-only access, DJBuilder for data modeling, and DJAdmin for system administration, as well as a Reflection Service for metadata synchronization and integration with Claude Code via MCP tools and skills.

Tokens
133.1K
Snippets
413
Records
566
Agent score
48%

What's inside DataJunction

  2. Overview of DataJunction plugin components

    main

    The DataJunction plugin for Claude Code consists of three main components:

    • skills/: Contextual DataJunction skills covering core concepts, querying, semantic modeling, repo-backed authoring, and the REST API. Claude loads these automatically based on the task.
    • agents/dj.md: A specialized DJ subagent with pre-loaded skills, providing DataJunction expertise without manual invocation.
    • .mcp.json: Configuration for the dj-mcp server, which provides the bridge between Claude and a live DataJunction instance for operational tasks like SQL generation and lineage.

  3. Ways to perform data modeling in DataJunction

    main

    Data modeling in DataJunction (DJ) can be accomplished through several interfaces depending on your workflow requirements:

    • DataJunction UI: For visual modeling and management.
    • DJ Server API: For programmatic control and automation.
    • Supported Clients: Using official DJ client libraries (e.g., Python client).
    • YAML files: For declarative, version-controlled modeling definitions.

  4. What is the DataJunction Reflection Service (DJRS)?

    main

    The DataJunction Reflection Service (DJRS) is a decoupled service that enables live table reflection within the DJ server. DJ uses an OpenAPI specification-based plugin design to interact with this service.

    Key components:

    • DJRS (Reflection Service): An open-source service that can be utilized by the DJ server to manage schema synchronization.
    • DJQS (Query Service): An open-source query service that implements the optional endpoints required by DJRS.

    Note for Custom Implementations: If you are running a custom query service instead of DJQS, you only need to implement the specific optional endpoints defined in the OpenAPI spec if you want to enable reflection capabilities.

  5. The Data Modeling workflow stages

    main

    The standard lifecycle for onboarding and building data models in DataJunction follows these stages:

    1. Create Namespaces: Organize your data models into logical namespaces.
    2. Register Source Nodes: Register your raw tables as source nodes in DJ.
    3. Create Transform Nodes: Define transform nodes for any lightweight SQL transformations required.
    4. Create Dimension Nodes: Define dimension nodes to represent the dimensions needed for analysis. These can be simple references to existing source nodes if modeling was done externally.
    5. Link Dimensions: Establish dimension links between your source, transform, or dimension nodes to construct the dimensional graph.
    6. Create Metric Nodes: Define metric nodes, which are aggregate expressions calculated over fields from source, transform, or dimension nodes.
    7. Create Cube Nodes: Bundle frequently used metrics and dimensions into cube nodes. Cubes are often used to facilitate materialization for high-performance access, such as powering analytical dashboards.

  6. How dimension nodes work in DataJunction

    main

    In DataJunction (DJ), a dimension node is a named node containing a query that selects from one or more other nodes. While similar to transform nodes, dimension nodes are unique because they can be referenced by columns on any source node, transform node, or other dimension nodes.

    Instead of manually performing joins to enrich data, you can tag specific columns with information about which dimension node they are tied to. This allows DJ to:

    1. Discover attributes: When asked what dimension attributes are available for a specific metric, DJ uses these labels to list all attribute columns from the associated dimension node.
    2. Automate joins: When a user requests a metric grouped by dimension attributes (e.g., grouping repair orders by dispatcher contact info), DJ uses the column label to automatically perform the correct join and retrieve the extra attributes.

    This abstraction allows users to interact with data via an intuitive API without needing to know the underlying table layouts or join keys.

  8. Decomposition patterns for common aggregations

    main

    DataJunction handles different aggregation types using specific decomposition patterns:

    Simple Aggregations

    SUM, COUNT, MIN, and MAX are already additive and decompose trivially.

    AVG (Average)

    Requires two components: sum_col and count_col.

    • Accumulate: SUM(x) and COUNT(x)
    • Merge: Both use SUM to roll up.
    • Combiner: SUM(sum_col) / SUM(count_col)

    APPROX_COUNT_DISTINCT (HyperLogLog)

    Uses probabilistic HLL sketches that can be merged.

    • Accumulate: hll_sketch_agg(x)
    • Merge: hll_union
    • Combiner: hll_sketch_estimate(hll_union(hll_col))

    Note: DJ uses Spark function names internally and translates them to target dialects like Druid or Trino at query time.

    Variance & Standard Deviation

    Uses the identity VAR = E[X²] - E[X]² and requires three components:

    • sum_col (SUM(x))
    • sum_sq_col (SUM(POWER(x, 2)))
    • count_col (COUNT(x))

    Covariance & Correlation

    • Covariance requires: sum_x, sum_y, sum_xy, and count.
    • Correlation requires 6 components: sum_x, sum_y, sum_x_sq, sum_y_sq, sum_xy, and count.
    -- Example: Decomposed measures for STDDEV(response_time)
SELECT 
  SUM(response_time) AS response_time_sum,
  SUM(POWER(response_time, 2)) AS response_time_sum_sq,
  COUNT(response_time) AS response_time_count
FROM requests

  9. Understand the DataJunction Repo-Backed Workflow

    main

    The repo-backed workflow allows you to author DataJunction (DJ) node definitions using YAML files stored in a git repository. This approach provides version control, pull request review workflows, branch-based development (for feature testing or environments), and a declarative 'infrastructure as code' model for your semantic layer.

    When using this workflow, node definitions are stored as YAML files, and changes are deployed via CI/CD pipelines following a PR-driven flow.

  10. How the three DataJunction client classes work together

    main

    DataJunction provides three distinct client classes depending on your intended task:

    1. DJClient: Use this for read-only access. It allows you to explore metrics, dimensions, SQL, and retrieve data.
    2. DJBuilder: Use this for data modelling. It allows you to modify the DJ data model, create new nodes (sources, dimensions, metrics, etc.), and manage namespaces or tags.
    3. DJAdmin: Use this for system administration. It is used to define connections to your data catalogs and query engines.

  11. How Git integration works in DataJunction

    main

    DataJunction (DJ) allows you to link a namespace to a Git repository to enable version-controlled workflows for managing node definitions. You can operate in two primary modes:

    1. Git as source of truth: By enabling git-only, the linked namespace becomes read-only in the UI. All changes must be made via Git commits and merged into the repository's main branch.
    2. UI-driven development: You can create branches directly from the DJ UI. This creates a new Git branch and a corresponding DJ namespace (e.g., root.namespace.branch_name) containing copies of the nodes from the parent namespace. You can then edit nodes in the UI, commit them to the Git branch, and open Pull Requests (PRs) to merge them back to the main branch.

  12. Understand Query Routing and Aggregate Awareness

    main

    Query routing is the process by which DataJunction (DJ) automatically decides whether to read a metric query from a pre-aggregated (materialized) dataset or to compute it live from source tables.

    This automatic selection is called aggregate awareness. When a suitable materialization exists, DJ routes the query to it without the user needing to specify a particular cube or pre-aggregation. DJ always picks the most efficient source: the coarsest available aggregate that still covers the requested dimensions and metrics.

    Key Routing Logic:

    • Roll-ups: If a request asks for total_revenue by region, but an aggregate exists at order_date × region, DJ will roll up the aggregate to satisfy the request.
    • Live Fallback: If a requested dimension (e.g., product) is not present in any available aggregates, DJ falls back to computing the query live from source tables.
    • Selection Priority: When multiple aggregates qualify, DJ selects the one with the fewest dimensions to minimize the amount of data that needs to be rolled up.